Oct 30 22:15:47.991099 ip-10-0-138-251 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 22:15:47.991110 ip-10-0-138-251 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 22:15:47.991117 ip-10-0-138-251 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 22:15:47.991404 ip-10-0-138-251 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 22:15:58.072532 ip-10-0-138-251 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 22:15:58.072550 ip-10-0-138-251 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot aec00bea8e2a4cad9a259de3830471a6 --
Oct 30 22:17:57.495796 ip-10-0-138-251 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:17:57.902881 ip-10-0-138-251 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:17:57.902881 ip-10-0-138-251 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:17:57.902881 ip-10-0-138-251 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:17:57.902881 ip-10-0-138-251 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:17:57.907534 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907431    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:17:57.907534 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907533    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:17:57.907534 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907536    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:17:57.907534 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907540    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907543    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907547    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907550    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907552    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907555    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907558    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907561    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907563    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907566    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907568    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907571    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907575    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907578    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907581    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907584    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907586    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907589    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907592    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:17:57.907654 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907596    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907600    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907603    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907607    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907610    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907613    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907616    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907618    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907622    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907625    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907627    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907630    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907633    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907637    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907640    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907643    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907646    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907649    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907652    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907655    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:17:57.908119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907658    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907661    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907664    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907666    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907669    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907672    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907676    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907680    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907683    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907686    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907689    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907691    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907694    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907696    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907699    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907702    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907704    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907707    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:17:57.908600 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907709    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907712    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907715    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907718    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907721    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907724    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907727    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907730    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907733    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907735    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907738    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907741    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907743    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907746    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907748    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907751    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907754    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907756    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907759    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907761    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:17:57.909057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907764    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907766    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907769    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907771    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907774    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907776    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907779    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907781    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907784    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907786    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907789    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907792    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907794    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907798    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907801    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907803    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907806    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907809    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907812    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907814    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:17:57.909558 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907818    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:17:57.910025 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907821    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:17:57.910025 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907823    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:17:57.910025 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907826    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:17:57.910025 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907829    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:17:57.910025 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.907831    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:17:57.910280 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910263    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:17:57.910280 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910279    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910284    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910287    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910291    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910294    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910296    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910299    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910301    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910305    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910308    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910310    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910313    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910315    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910318    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910321    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910323    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910326    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910329    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910332    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910335    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:17:57.910333 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910338    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910341    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910343    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910346    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910349    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910351    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910355    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910361    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910364    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910366    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910369    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910372    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910375    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910378    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910381    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910383    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910386    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910388    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910391    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:17:57.910835 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910394    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910396    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910399    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910402    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910404    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910407    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910409    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910412    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910414    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910417    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910419    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910422    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910424    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910426    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910429    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910432    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910436    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910438    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910441    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910444    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:17:57.911292 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910446    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910449    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910451    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910454    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910457    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910459    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910462    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910465    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910467    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910470    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910473    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910475    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910478    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910480    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910483    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910488    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910492    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910496    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910499    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:17:57.911778 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910501    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910519    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910523    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910528    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910532    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910535    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910538    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910541    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910544    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910547    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910550    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910552    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910555    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910558    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910561    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910564    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910566    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910569    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910571    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:17:57.912214 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910575    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:17:57.912677 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910578    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:17:57.912677 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910581    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:17:57.912677 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910584    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:17:57.912677 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910586    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:17:57.912677 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910589    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:17:57.912677 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910592    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:17:57.912677 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.910594    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:17:57.913216 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913093    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:17:57.913216 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913215    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913221    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913226    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913230    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913235    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913239    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913246    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913253    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913265    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913270    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913281    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913285    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913289    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913294    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913298    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913302    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913307    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913311    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913314    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:17:57.913329 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913318    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913323    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913329    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913341    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913346    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913351    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913355    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913359    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913363    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913367    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913371    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913376    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913380    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913413    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913417    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913421    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913430    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913434    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913439    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913443    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:17:57.913973 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913447    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913453    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913458    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913462    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913466    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913470    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913474    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913483    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913488    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913492    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913497    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913501    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913521    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913526    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913531    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913535    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913539    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913543    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913547    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913552    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:17:57.914441 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913561    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913565    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913569    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913573    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913577    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913581    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913612    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913616    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913621    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913625    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913630    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913634    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913638    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913647    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913652    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913656    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913660    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913664    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913669    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:17:57.914971 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913673    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913677    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913681    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913685    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913689    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913693    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913697    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913708    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913713    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913717    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913722    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913726    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913730    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913734    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913739    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913745    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913749    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913754    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913758    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:17:57.915406 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913822    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913848    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913854    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913858    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913861    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913865    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913867    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.913874    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915165    2583 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915179    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915186    2583 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915191    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915196    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915199    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915204    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915208    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915211    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915214    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915218    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915221    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915225    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915228    2583 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915231    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:17:57.915863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915234    2583 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915236    2583 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915239    2583 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915244    2583 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915247    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915250    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915253    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915257    2583 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915260    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915264    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915267    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915271    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915275    2583 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915278    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915281    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915284    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915287    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915291    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915294    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915297    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915300    2583 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915303    2583 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915306    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915311    2583 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915314    2583 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:17:57.916426 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915317    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915320    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915323    2583 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915328    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915330    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915334    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915337    2583 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915340    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915343    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915346    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915349    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915352    2583 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915355    2583 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915357    2583 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915362    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915366    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915369    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915372    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915376    2583 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915379    2583 flags.go:64] FLAG: --help="false"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915382    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915385    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915388    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:17:57.917059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915391    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915396    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915400    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915402    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915405    2583 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915408    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915411    2583 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915415    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915418    2583 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915421    2583 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915424    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915427    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915430    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915434    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915437    2583 flags.go:64] FLAG: --lock-file=""
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915440    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915443    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915446    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915451    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915454    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915457    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915460    2583 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915463    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915467    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:17:57.917719 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915470    2583 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915472    2583 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915478    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915481    2583 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915485    2583 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915488    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915491    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915494    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915497    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915500    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915503    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915516    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915527    2583 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915530    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915533    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915536    2583 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915539    2583 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915542    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915545    2583 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915548    2583 flags.go:64] FLAG: --port="10250"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915551    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915554    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-078c84db754bfab61"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915558    2583 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915561    2583 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915564    2583 flags.go:64] FLAG: --register-node="true"
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915566    2583 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:17:57.918293 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915571    2583 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915574    2583 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915577    2583 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915579    2583 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915583    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915586    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915589    2583 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915593    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915596    2583 flags.go:64] FLAG: --runonce="false"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915598    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915602    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915605    2583 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915608    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915610    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915614    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915617    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915620    2583 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915622    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915625    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915628    2583 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915631    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915634    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915638    2583 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915641    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915647    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:17:57.918922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915650    2583 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915670    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915676    2583 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915679    2583 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915684    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915687    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915691    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915694    2583 flags.go:64] FLAG: --v="2"
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915698    2583 flags.go:64] FLAG: --version="false"
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915703    2583 flags.go:64] FLAG: --vmodule=""
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915708    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.915711    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915870    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915874    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915877    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915881    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915885    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915889    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915892    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915898    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915901    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915904    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:17:57.919487 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915906    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915909    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915912    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915915    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915917    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915920    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915922    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915925    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915928    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915931    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915933    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915936    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915938    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915941    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915944    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915947    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915949    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915952    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915955    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915958    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:17:57.920057 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915961    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915963    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915966    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915968    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915971    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915973    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915976    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915978    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915981    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915985    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915988    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915991    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915993    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915996    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.915998    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916001    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916004    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916006    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916009    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:17:57.920615 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916012    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916014    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916017    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916019    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916022    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916024    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916027    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916030    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916033    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916036    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916040    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916043    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916046    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916048    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916051    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916054    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916057    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916059    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916062    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916065    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:17:57.921065 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916067    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916070    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916073    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916076    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916079    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916082    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916084    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916087    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916090    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916092    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916095    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916098    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916100    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916103    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916105    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916108    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916111    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916114    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916116    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916119    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:17:57.921561 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916121    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916124    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916128    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916131    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916133    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916136    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916139    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916142    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916144    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916148    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916152    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916155    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916158    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916161    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916165    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916167    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:17:57.922042 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916170    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.916196    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916234    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916239    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916241    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916244    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916247    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916250    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916252    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916255    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916258    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916260    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916263    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916266    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916268    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916271    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:17:57.922418 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916274    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916276    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916280    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916289    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916292    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916294    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916297    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916300    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916302    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916305    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916307    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916310    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916313    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916315    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916318    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916322    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916325    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916327    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916330    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916333    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:17:57.922804 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916335    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916338    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916341    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916343    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916346    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916348    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916351    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916354    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916356    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916359    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916361    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916364    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916366    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916369    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916372    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916375    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916379    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916381    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916384    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916386    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:17:57.923261 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916389    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916392    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916394    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916397    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916399    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916402    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916404    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916409    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916411    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916414    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916417    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916419    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916422    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916424    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916427    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916430    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916432    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916435    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916438    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916441    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:17:57.923761 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916444    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916446    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916449    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916452    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916454    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916456    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916460    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916462    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916465    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916468    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916471    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916473    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916476    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916480    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916483    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916486    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916489    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916491    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916494    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:17:57.924222 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916497    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916500    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916502    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916517    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916519    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916523    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916527    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916531    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916534    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916537    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916540    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916542    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.916545    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.924085    2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.924098    2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.924118    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.924122    2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924183    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924188    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:17:57.924732 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924191    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924194    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924197    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924200    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924203    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924206    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924209    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924212    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924215    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924217    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924220    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924223    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924226    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924229    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924232    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924235    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924237    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924240    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924243    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:17:57.925207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924246    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924248    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924251    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924253    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924256    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924259    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924261    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924264    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924268    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924271    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924274    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924276    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924279    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924281    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924284    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924286    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924289    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924292    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924294    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924297    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:17:57.926277 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924300    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924302    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924305    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924307    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924310    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924313    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924316    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924318    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924321    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924323    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924326    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924329    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924331    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924334    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924336    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924339    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924341    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924344    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924346    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924349    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:17:57.926759 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924352    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924358    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924361    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924364    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924366    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924369    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924372    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924375    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924377    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924380    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924383    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924385    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924388    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924390    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924393    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924395    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924398    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924401    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924405    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:17:57.927225 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924409    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924412    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924416    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924418    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924421    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924424    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924427    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924429    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924432    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924435    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924437    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924440    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924443    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924445    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924449    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924453    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924456    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924459    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924462    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:17:57.927684 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924464    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924467    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924469    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924472    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924475    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924477    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924480    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.924518    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924667    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924672    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924676    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924679    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924682    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924685    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924687    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924691    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:17:57.928119 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924694    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924697    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924700    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924703    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924705    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924708    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924710    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924713    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924717    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924720    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924723    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924725    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924728    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924731    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924734    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924737    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924739    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924742    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924745    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:17:57.928486 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924748    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924751    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924753    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924756    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924758    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924761    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924763    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924766    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924769    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924771    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924774    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924776    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924779    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924782    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924784    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924787    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924789    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924792    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924794    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924797    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:17:57.928944 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924799    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924802    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924804    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924807    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924810    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924812    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924815    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924818    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924821    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924823    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924826    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924829    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924831    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924834    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924836    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924839    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924841    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924844    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924847    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924849    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:17:57.929448 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924852    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924855    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924857    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924860    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924862    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924865    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924868    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924870    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924873    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924875    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924878    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924880    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924883    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924885    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924888    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924890    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924893    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924895    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924898    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:17:57.929933 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924900    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924903    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924909    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924911    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924914    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924916    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924919    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924922    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924924    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924927    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924929    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924932    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924935    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924937    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924940    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924942    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924945    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924947    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924950    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:17:57.930369 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.924953    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.924976    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925010    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925013    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925017    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925020    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925023    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925026    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925029    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925031    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925034    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925036    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925039    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925041    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925044    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925047    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:17:57.930834 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925052    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925054    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925057    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925060    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925063    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925065    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925068    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925070    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925073    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925076    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925078    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925081    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925083    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925086    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925088    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925091    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925094    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925097    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925099    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:17:57.931207 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925102    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925104    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925107    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925110    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925113    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925115    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925118    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925120    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925123    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925126    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925129    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925133    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925137    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925141    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925144    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925148    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925151    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925154    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925156    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:17:57.931667 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925160    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925164    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925168    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925171    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925173    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925176    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925179    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925182    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925184    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925187    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925189    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925192    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925194    2583 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925197    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925200    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925202    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925205    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925207    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925210    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925213    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:17:57.932100 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925216    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925218    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925221    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925223    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925226    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925229    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925233    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925236    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925243    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925246    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925249    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925251    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925254    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925264    2583 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925268    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925270    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925273    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925276    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925278    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925281    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:17:57.932669 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925284    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925287    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925289    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925292    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925294    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925297    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925300    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925302    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925306    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925309    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925311    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925314    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925316    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:57.925319    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.925379    2583 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.927363    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.928306    2583 server.go:1014] "Starting client certificate rotation"
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.928400    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:17:57.933137 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.928448    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:17:57.933602 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.930360    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:17:57.933602 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.931751    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:17:57.940357 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.940336    2583 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:17:57.943852 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.943835    2583 log.go:25] "Validated CRI v1 image API"
Oct 30 22:17:57.944703 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.944671    2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:17:57.949662 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.949643    2583 fs.go:135] Filesystem UUIDs: map[6c422d92-e786-44e4-922d-e390a0764faf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8948b817-8f27-4388-8be1-7a7cdc16a4c5:/dev/nvme0n1p3]
Oct 30 22:17:57.949730 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.949662    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 22:17:57.955731 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.955621    2583 manager.go:217] Machine: {Timestamp:2026-10-30 22:17:57.953967077 +0000 UTC m=+0.341234919 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099415 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2603805aae9ef7962cc5589ad9da9c SystemUUID:ec260380-5aae-9ef7-962c-c5589ad9da9c BootID:aec00bea-8e2a-4cad-9a25-9de3830471a6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:b7:a2:1d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:b7:a2:1d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:73:7a:f6:6d:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:17:57.956321 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.956311    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:17:57.956422 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.956407    2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:17:57.958358 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.958338    2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:17:57.958489 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.958359    2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-251.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:17:57.958544 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.958491    2583 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:17:57.958544 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.958498    2583 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:17:57.958544 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.958531    2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:17:57.958544 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.958544    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:17:57.959318 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.959307    2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:17:57.962898 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.962881    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:17:57.962955 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.962887    2583 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:17:57.963050 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.963040    2583 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:17:57.963087 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.963072    2583 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:17:57.963087 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.963082    2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:17:57.963141 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.963092    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:17:57.963141 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.963101    2583 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:17:57.963141 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.963109    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:17:57.963692 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.963679    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:17:57.965841 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.965823    2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:17:57.969641 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.969627    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:17:57.970409 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.970396    2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:17:57.970450 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.970427    2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:17:57.970618 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.970607    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:17:57.970665 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.970627    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:17:57.970665 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.970637    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:17:57.970665 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.970647    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:17:57.970665 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.970656    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:17:57.970665 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.970664    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:17:57.971373 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.971361    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:17:57.971407 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.971375    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:17:57.971407 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.971383    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:17:57.971407 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.971389    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:17:57.971407 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.971398    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:17:57.971407 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.971407    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:17:57.972458 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.972448    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:17:57.972458 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.972459    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:17:57.976090 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.976076    2583 server.go:1263] "Started kubelet"
Oct 30 22:17:57.976265 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.976132    2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:17:57.976349 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.976265    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:17:57.976491 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.976468    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:17:57.976861 ip-10-0-138-251 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:17:57.979638 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.979564    2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:17:57.981763 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.981742    2583 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:17:57.982738 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.982663    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-251.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 22:17:57.985301 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:57.982758    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-251.us-east-2.compute.internal.18e36e98856ad408  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-251.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-251.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 22:17:57.976044552 +0000 UTC m=+0.363312390,LastTimestamp:2026-10-30 22:17:57.976044552 +0000 UTC m=+0.363312390,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-251.us-east-2.compute.internal,}"
Oct 30 22:17:57.985690 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.985662    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:17:57.986633 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.986613    2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:17:57.987524 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:57.987103    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:57.987524 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.987282    2583 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:17:57.987524 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.987295    2583 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:17:57.987524 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.987421    2583 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:17:57.987524 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.987429    2583 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:17:57.987524 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.987527    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:17:57.987838 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:57.987691    2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 22:17:57.987838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.987763    2583 factory.go:55] Registering systemd factory
Oct 30 22:17:57.987838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.987776    2583 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:17:57.988028 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.988012    2583 factory.go:153] Registering CRI-O factory
Oct 30 22:17:57.988100 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.988033    2583 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:17:57.988100 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.988082    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:17:57.988190 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.988102    2583 factory.go:103] Registering Raw factory
Oct 30 22:17:57.988190 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.988121    2583 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:17:57.989064 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:57.989049    2583 manager.go:319] Starting recovery of all containers
Oct 30 22:17:57.991149 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:57.991120    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-251.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 22:17:57.991411 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:57.991385    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-251.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 22:17:57.991411 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:57.991393    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 22:17:58.000452 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.000436    2583 manager.go:324] Recovery completed
Oct 30 22:17:58.000589 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.000564    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 22:17:58.005881 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.005868    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:17:58.008005 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.007986    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:17:58.008073 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.008026    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:17:58.008073 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.008043    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:17:58.008597 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.008584    2583 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:17:58.008597 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.008595    2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:17:58.008690 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.008609    2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:17:58.008690 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.008583    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:17:58.009298 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.009280    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:17:58.009370 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.009321    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:17:58.009370 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.009337    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:17:58.010562 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.010467    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-251.us-east-2.compute.internal.18e36e9887528330  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-251.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-251.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-251.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 22:17:58.008005424 +0000 UTC m=+0.395273269,LastTimestamp:2026-10-30 22:17:58.008005424 +0000 UTC m=+0.395273269,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-251.us-east-2.compute.internal,}"
Oct 30 22:17:58.019071 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.019054    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qsj75"
Oct 30 22:17:58.026800 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.026724    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-251.us-east-2.compute.internal.18e36e988752f2c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-251.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-251.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-251.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 22:17:58.00803399 +0000 UTC m=+0.395301830,LastTimestamp:2026-10-30 22:17:58.00803399 +0000 UTC m=+0.395301830,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-251.us-east-2.compute.internal,}"
Oct 30 22:17:58.031312 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.031294    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qsj75"
Oct 30 22:17:58.044669 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.044606    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-251.us-east-2.compute.internal.18e36e9887532abc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-251.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-251.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-251.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 22:17:58.008048316 +0000 UTC m=+0.395316154,LastTimestamp:2026-10-30 22:17:58.008048316 +0000 UTC m=+0.395316154,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-251.us-east-2.compute.internal,}"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.061923    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-251.us-east-2.compute.internal.18e36e9887528330\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-251.us-east-2.compute.internal.18e36e9887528330  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-251.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-251.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-251.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 22:17:58.008005424 +0000 UTC m=+0.395273269,LastTimestamp:2026-10-30 22:17:58.009296391 +0000 UTC m=+0.396564229,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-251.us-east-2.compute.internal,}"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.062000    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.063080    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.063903    2583 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.063931    2583 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.063964    2583 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.075783    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-251.us-east-2.compute.internal.18e36e988752f2c6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-251.us-east-2.compute.internal.18e36e988752f2c6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-251.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-251.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-251.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 22:17:58.00803399 +0000 UTC m=+0.395301830,LastTimestamp:2026-10-30 22:17:58.009328742 +0000 UTC m=+0.396596581,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-251.us-east-2.compute.internal,}"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.080558    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-251.us-east-2.compute.internal.18e36e9887532abc\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-251.us-east-2.compute.internal.18e36e9887532abc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-251.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-251.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-251.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 22:17:58.008048316 +0000 UTC m=+0.395316154,LastTimestamp:2026-10-30 22:17:58.009342492 +0000 UTC m=+0.396610334,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-251.us-east-2.compute.internal,}"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.080848    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.087256    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.148539    2583 policy_none.go:50] "Start"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.148559    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:17:58.150319 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.148575    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:17:58.152834 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.150929    2583 policy_none.go:44] "Start"
Oct 30 22:17:58.164784 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.164769    2583 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 22:17:58.187569 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.187546    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:58.189100 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.189073    2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:17:58.193444 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.193422    2583 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:17:58.193527 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.193457    2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:17:58.193527 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.193469    2583 server.go:87] "Starting device plugin registration server"
Oct 30 22:17:58.193527 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.193502    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-251.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Oct 30 22:17:58.193650 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.193568    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-251.us-east-2.compute.internal.18e36e98923d1010  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-251.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-138-251.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 22:17:58.191149072 +0000 UTC m=+0.578416897,LastTimestamp:2026-10-30 22:17:58.191149072 +0000 UTC m=+0.578416897,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-251.us-east-2.compute.internal,}"
Oct 30 22:17:58.193722 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.193712    2583 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:17:58.193760 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.193723    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:17:58.194005 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.193857    2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:17:58.194005 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.193953    2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:17:58.194005 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.193962    2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:17:58.194379 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.194361    2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:17:58.194451 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.194413    2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:58.294401 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.294380    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:17:58.295365 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.295349    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:17:58.295439 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.295381    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:17:58.295439 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.295393    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:17:58.295439 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.295417    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.305859 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.305838    2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.305914 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.305859    2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-251.us-east-2.compute.internal\": node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:58.340523 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.340486    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:58.365608 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.365572    2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"]
Oct 30 22:17:58.365728 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.365662    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:17:58.366462 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.366447    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:17:58.366559 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.366475    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:17:58.366559 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.366488    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:17:58.368668 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.368655    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:17:58.368824 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.368809    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.368891 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.368874    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:17:58.369421 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.369404    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:17:58.369517 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.369432    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:17:58.369517 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.369437    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:17:58.369517 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.369461    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:17:58.369517 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.369472    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:17:58.369517 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.369443    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:17:58.371678 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.371661    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.371749 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.371689    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:17:58.372361 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.372338    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:17:58.372425 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.372371    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:17:58.372425 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.372388    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:17:58.388415 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.388396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5c727ac326546386befce51c4d99c6d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"a5c727ac326546386befce51c4d99c6d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.388533 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.388427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.388533 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.388451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.397152 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.397133    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found" node="ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.399781 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.399764    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found" node="ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.441463 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.441404    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:58.488824 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.488788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5c727ac326546386befce51c4d99c6d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"a5c727ac326546386befce51c4d99c6d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.488824 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.488821    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5c727ac326546386befce51c4d99c6d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"a5c727ac326546386befce51c4d99c6d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.489001 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.488836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.489001 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.488863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.489001 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.488900    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.489001 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.488906    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.541924 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.541887    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:58.642333 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.642302    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:58.699829 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.699764    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.702721 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.702703    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:58.742609 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.742576    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:58.843089 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.843052    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:58.929485 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:58.929452    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 22:17:58.943742 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:58.943712    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:59.040602 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.040496    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:17:59.041562 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.041532    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:17:59.041690 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.041577    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:17:59.041690 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.041597    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:17:59.043807 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:59.043785    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:17:59.067641 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.067607    2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:17:59.087604 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.087565    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:59.104436 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.104414    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:17:59.106376 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.106359    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:17:59.116593 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.116570    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:17:59.150801 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.150766    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 22:12:58 +0000 UTC" deadline="2028-07-08 10:55:04.448527742 +0000 UTC"
Oct 30 22:17:59.150801 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.150795    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14796h37m5.297735451s"
Oct 30 22:17:59.163114 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.163086    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:17:59.229371 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.229337    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:17:59.254748 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.254577    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:17:59.436442 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:59.436409    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65deeb7675a6280ac03893337db51c32.slice/crio-3ebc14ac21416b778201bd1361cb164993cb192ee203cbc22c7419765d5202ce WatchSource:0}: Error finding container 3ebc14ac21416b778201bd1361cb164993cb192ee203cbc22c7419765d5202ce: Status 404 returned error can't find the container with id 3ebc14ac21416b778201bd1361cb164993cb192ee203cbc22c7419765d5202ce
Oct 30 22:17:59.436905 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:17:59.436889    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c727ac326546386befce51c4d99c6d.slice/crio-94be0e38e90de037403a5baf5200eab1e6a6d12380ad5ebd875bf62f85d7d53c WatchSource:0}: Error finding container 94be0e38e90de037403a5baf5200eab1e6a6d12380ad5ebd875bf62f85d7d53c: Status 404 returned error can't find the container with id 94be0e38e90de037403a5baf5200eab1e6a6d12380ad5ebd875bf62f85d7d53c
Oct 30 22:17:59.442117 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.442101    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:17:59.965938 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.965904    2583 apiserver.go:52] "Watching apiserver"
Oct 30 22:17:59.974979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.974948    2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:17:59.975285 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.975268    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal","openshift-multus/multus-kfshl","openshift-ovn-kubernetes/ovnkube-node-7ppff","kube-system/global-pull-secret-syncer-xkt2l","kube-system/konnectivity-agent-wb9bv","openshift-dns/node-resolver-qqwss","openshift-multus/network-metrics-daemon-hmgsj","openshift-network-operator/iptables-alerter-vhkmj","openshift-cluster-node-tuning-operator/tuned-ggnd6","openshift-image-registry/node-ca-7cjdl","openshift-multus/multus-additional-cni-plugins-64cvm","openshift-network-diagnostics/network-check-target-tgn5k","kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"]
Oct 30 22:17:59.979175 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.979159    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:17:59.981498 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.981477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:17:59.983835 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.983812    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:17:59.983946 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.983834    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:17:59.983946 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.983870    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:17:59.983946 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.983908    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jb8sb\""
Oct 30 22:17:59.984609 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.984589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.985518 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.985451    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:17:59.985518 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.985496    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:17:59.985518 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.985496    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:17:59.985701 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.985497    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:17:59.985701 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.985542    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:17:59.985701 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.985504    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kkwnr\""
Oct 30 22:17:59.985913 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.985900    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 22:17:59.987016 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.986998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:17:59.988285 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.988262    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:17:59.988585 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.988565    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:17:59.988675 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.988609    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:17:59.988675 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.988613    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:17:59.988762 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.988703    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jm6gb\""
Oct 30 22:17:59.988762 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.988740    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:17:59.989514 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.989476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:17:59.989555 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.989503    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:17:59.990611 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.990589    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:17:59.990881 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.990866    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-szxgd\""
Oct 30 22:17:59.990974 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.990955    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:17:59.991028 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.990961    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:17:59.995028 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.992562    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:17:59.995336 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.995312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:17:59.996683 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.996666    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jmj7l\""
Oct 30 22:17:59.997734 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.997716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:17:59.997822 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.997741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-node-log\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.997822 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.997758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-device-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:17:59.997822 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.997778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tdx8\" (UniqueName: \"kubernetes.io/projected/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-kube-api-access-7tdx8\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:17:59.997822 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.997797    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-kubelet\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.997979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.997852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-env-overrides\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.997979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.997902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-script-lib\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.997979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.997944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-socket-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:17:59.998072 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.997983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-host\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:17:59.998072 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:17:59.998072 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-systemd-units\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998072 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-slash\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998252 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-systemd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998252 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998252 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998170    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-netd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998252 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:17:59.998252 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998232    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-system-cni-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:17:59.998516 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:17:59.998516 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-var-lib-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998516 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998330    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:17:59.998516 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-config\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998516 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:17:59.998430    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:17:59.998516 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60457f9c-65b0-4547-9012-2fc4efe2b9e8-konnectivity-ca\") pod \"konnectivity-agent-wb9bv\" (UID: \"60457f9c-65b0-4547-9012-2fc4efe2b9e8\") " pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:17:59.998516 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-cnibin\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:17:59.998516 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998499    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-netns\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-log-socket\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvr4\" (UniqueName: \"kubernetes.io/projected/b40f9c33-c4fe-4176-9ae9-561592a770f6-kube-api-access-tgvr4\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-bin\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovn-node-metrics-cert\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkrjv\" (UniqueName: \"kubernetes.io/projected/9af08ecc-3199-4c8b-b364-f42f4befeff1-kube-api-access-nkrjv\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60457f9c-65b0-4547-9012-2fc4efe2b9e8-agent-certs\") pod \"konnectivity-agent-wb9bv\" (UID: \"60457f9c-65b0-4547-9012-2fc4efe2b9e8\") " pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-serviceca\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-ovn\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-sys-fs\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zptr\" (UniqueName: \"kubernetes.io/projected/de0170e7-99d6-4102-9b88-13552cb6f421-kube-api-access-2zptr\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:17:59.998979 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.998979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-etc-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:17:59.999617 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.999006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-registration-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:17:59.999617 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:17:59.999024    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-etc-selinux\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.001003 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.000985    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:00.001090 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.001063    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:00.002679 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.002655    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:18:00.003308 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.003288    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.005710 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.005689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.007896 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.007879    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:00.007970 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.007930    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:00.031871 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.031841    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kljdg"
Oct 30 22:18:00.048675 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.048654    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kljdg"
Oct 30 22:18:00.071203 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.070744    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal" event={"ID":"65deeb7675a6280ac03893337db51c32","Type":"ContainerStarted","Data":"3ebc14ac21416b778201bd1361cb164993cb192ee203cbc22c7419765d5202ce"}
Oct 30 22:18:00.072717 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.072692    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" event={"ID":"a5c727ac326546386befce51c4d99c6d","Type":"ContainerStarted","Data":"94be0e38e90de037403a5baf5200eab1e6a6d12380ad5ebd875bf62f85d7d53c"}
Oct 30 22:18:00.089368 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.089343    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:18:00.094955 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.094927    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:00.095126 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095085    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:18:00.095191 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095114    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:18:00.095249 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095224    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:18:00.095301 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095291    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4x7f7\""
Oct 30 22:18:00.095430 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095415    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:00.095621 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095598    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-glgqx\""
Oct 30 22:18:00.095715 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095656    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tp94j\""
Oct 30 22:18:00.095797 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095777    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:18:00.095906 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095841    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:00.095906 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095896    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:18:00.096031 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095950    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:00.096031 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.095950    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qwcf4\""
Oct 30 22:18:00.096031 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.096007    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:18:00.099217 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-bin\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.099332 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099254    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovn-node-metrics-cert\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.099332 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-var-lib-kubelet\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.099449 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099373    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-bin\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.099449 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-serviceca\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:00.099449 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-ovn\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.099643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-sys-fs\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.099643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099494    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-ovn\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.099643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zptr\" (UniqueName: \"kubernetes.io/projected/de0170e7-99d6-4102-9b88-13552cb6f421-kube-api-access-2zptr\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.099643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgv2b\" (UniqueName: \"kubernetes.io/projected/012d43c7-e3f2-4931-877e-cdf66eaf61b6-kube-api-access-sgv2b\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.099643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099591    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-sys-fs\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.099643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099614    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-etc-selinux\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.099942 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:00.099942 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-host\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.099942 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.099942 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-cni-binary-copy\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.100153 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-etc-kubernetes\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.100153 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099962    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-etc-selinux\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.100153 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.099981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-socket-dir-parent\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.100525 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.100486    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-serviceca\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:00.100625 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.100560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tdx8\" (UniqueName: \"kubernetes.io/projected/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-kube-api-access-7tdx8\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:00.100625 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.100607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-env-overrides\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.100843 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.100812    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.101120 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5gx\" (UniqueName: \"kubernetes.io/projected/92a4974a-3e6e-4575-b5bd-303beb3fa201-kube-api-access-jp5gx\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:00.101212 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqzxg\" (UniqueName: \"kubernetes.io/projected/428997c4-b34e-4e3c-8551-c2d8510c69f4-kube-api-access-kqzxg\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:00.101212 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-netns\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.101333 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-conf-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.101333 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101225    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-env-overrides\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.101333 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-host\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:00.101490 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-systemd-units\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.101490 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101455    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-slash\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.101490 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101480    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-systemd-units\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.101735 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101484    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-host\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:00.101735 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101541    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-slash\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.101735 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-systemd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.101735 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101584    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-systemd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.101735 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-netd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.101735 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101678    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-netd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.101735 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.101735 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101684    2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-daemon-config\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101791    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-tuned\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-system-cni-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101922    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-system-cni-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101973    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-var-lib-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.101998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-config\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60457f9c-65b0-4547-9012-2fc4efe2b9e8-konnectivity-ca\") pod \"konnectivity-agent-wb9bv\" (UID: \"60457f9c-65b0-4547-9012-2fc4efe2b9e8\") " pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-k8s-cni-cncf-io\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102070    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-var-lib-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-hostroot\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.102139 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-netns\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-sys\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-tmp\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-kubelet-config\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-dbus\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvr4\" (UniqueName: \"kubernetes.io/projected/b40f9c33-c4fe-4176-9ae9-561592a770f6-kube-api-access-tgvr4\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrjv\" (UniqueName: \"kubernetes.io/projected/9af08ecc-3199-4c8b-b364-f42f4befeff1-kube-api-access-nkrjv\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60457f9c-65b0-4547-9012-2fc4efe2b9e8-agent-certs\") pod \"konnectivity-agent-wb9bv\" (UID: \"60457f9c-65b0-4547-9012-2fc4efe2b9e8\") " pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102427    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-netns\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-multus\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92a4974a-3e6e-4575-b5bd-303beb3fa201-hosts-file\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a4974a-3e6e-4575-b5bd-303beb3fa201-tmp-dir\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102588    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-etc-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.102838 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102689    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-etc-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-registration-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-bin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102739    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-registration-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm75h\" (UniqueName: \"kubernetes.io/projected/87c8cdc5-be2e-47a0-8cde-e944fda51056-kube-api-access-wm75h\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-run\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-node-log\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-device-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-lib-modules\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-kubelet\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102946    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102946    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.102959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-script-lib\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-socket-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-kubernetes\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-conf\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103109    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:00.103677 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103083    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-config\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103174    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-socket-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/407cf470-1b82-424c-a8e2-a5f061b3b84c-iptables-alerter-script\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103282    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysconfig\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103297    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-node-log\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-kubelet\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-multus-certs\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103247    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-device-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103424    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103455    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-script-lib\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-systemd\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103501    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-kubelet\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-cnibin\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-log-socket\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.104492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.105367 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/407cf470-1b82-424c-a8e2-a5f061b3b84c-host-slash\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:00.105367 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sfkt\" (UniqueName: \"kubernetes.io/projected/407cf470-1b82-424c-a8e2-a5f061b3b84c-kube-api-access-7sfkt\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:00.105367 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103764    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.105367 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-system-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.105367 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103813    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-cnibin\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.105367 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103847    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-log-socket\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.105367 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103877    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.105367 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-cnibin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.105367 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-modprobe-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.105367 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.103975    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.105922 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.105531    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60457f9c-65b0-4547-9012-2fc4efe2b9e8-konnectivity-ca\") pod \"konnectivity-agent-wb9bv\" (UID: \"60457f9c-65b0-4547-9012-2fc4efe2b9e8\") " pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:00.106866 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.106829    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovn-node-metrics-cert\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.109412 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.109388    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60457f9c-65b0-4547-9012-2fc4efe2b9e8-agent-certs\") pod \"konnectivity-agent-wb9bv\" (UID: \"60457f9c-65b0-4547-9012-2fc4efe2b9e8\") " pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:00.119836 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.119810    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvr4\" (UniqueName: \"kubernetes.io/projected/b40f9c33-c4fe-4176-9ae9-561592a770f6-kube-api-access-tgvr4\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.119977 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.119952    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrjv\" (UniqueName: \"kubernetes.io/projected/9af08ecc-3199-4c8b-b364-f42f4befeff1-kube-api-access-nkrjv\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.120515 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.120486    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tdx8\" (UniqueName: \"kubernetes.io/projected/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-kube-api-access-7tdx8\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:00.120984 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.120959    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zptr\" (UniqueName: \"kubernetes.io/projected/de0170e7-99d6-4102-9b88-13552cb6f421-kube-api-access-2zptr\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.204434 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.204682 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-multus\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.204682 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92a4974a-3e6e-4575-b5bd-303beb3fa201-hosts-file\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:00.204682 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a4974a-3e6e-4575-b5bd-303beb3fa201-tmp-dir\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:00.204682 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-bin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.204682 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm75h\" (UniqueName: \"kubernetes.io/projected/87c8cdc5-be2e-47a0-8cde-e944fda51056-kube-api-access-wm75h\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.204682 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-run\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.204682 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-lib-modules\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.204682 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-kubernetes\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.204682 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-conf\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/407cf470-1b82-424c-a8e2-a5f061b3b84c-iptables-alerter-script\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysconfig\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-kubelet\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-multus-certs\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-systemd\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204875    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/407cf470-1b82-424c-a8e2-a5f061b3b84c-host-slash\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sfkt\" (UniqueName: \"kubernetes.io/projected/407cf470-1b82-424c-a8e2-a5f061b3b84c-kube-api-access-7sfkt\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-system-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-cnibin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.204982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-modprobe-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-var-lib-kubelet\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgv2b\" (UniqueName: \"kubernetes.io/projected/012d43c7-e3f2-4931-877e-cdf66eaf61b6-kube-api-access-sgv2b\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-host\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.205201 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205136    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-cni-binary-copy\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-etc-kubernetes\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-socket-dir-parent\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5gx\" (UniqueName: \"kubernetes.io/projected/92a4974a-3e6e-4575-b5bd-303beb3fa201-kube-api-access-jp5gx\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzxg\" (UniqueName: \"kubernetes.io/projected/428997c4-b34e-4e3c-8551-c2d8510c69f4-kube-api-access-kqzxg\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-netns\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-conf-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-daemon-config\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-tuned\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-k8s-cni-cncf-io\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-hostroot\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-sys\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-tmp\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-kubelet-config\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-dbus\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205667    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-cnibin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205844    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.208553 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205887    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-var-lib-kubelet\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205923    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-modprobe-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205963    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-multus\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.206006    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92a4974a-3e6e-4575-b5bd-303beb3fa201-hosts-file\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.206341    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a4974a-3e6e-4575-b5bd-303beb3fa201-tmp-dir\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.206380    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-host\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.206432    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-bin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.206713    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-run\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.206842    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-lib-modules\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.206933    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-kubernetes\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.206982    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-cni-binary-copy\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.207028    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-etc-kubernetes\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.207065    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-conf\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.207079    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-socket-dir-parent\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.207144    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.207210    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret podName:adb5dbb9-f122-43fc-9ef9-867e6b2bb700 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:00.707190062 +0000 UTC m=+3.094457913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret") pod "global-pull-secret-syncer-xkt2l" (UID: "adb5dbb9-f122-43fc-9ef9-867e6b2bb700") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.207453    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-netns\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.207502    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-conf-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.209465 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.207973    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-daemon-config\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208405    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-systemd\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.208442    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208482    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-k8s-cni-cncf-io\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208496    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-system-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208496    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-kubelet\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208441    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208441    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysconfig\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.208512    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs podName:428997c4-b34e-4e3c-8551-c2d8510c69f4 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:00.708490462 +0000 UTC m=+3.095758303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs") pod "network-metrics-daemon-hmgsj" (UID: "428997c4-b34e-4e3c-8551-c2d8510c69f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208540    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-hostroot\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208541    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-multus-certs\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208564    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/407cf470-1b82-424c-a8e2-a5f061b3b84c-host-slash\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.205844    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-dbus\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208646    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-kubelet-config\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:00.210564 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.208743    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-sys\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.211690 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.211460    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-tmp\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.211690 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.211650    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/407cf470-1b82-424c-a8e2-a5f061b3b84c-iptables-alerter-script\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:00.211828 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.211792    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-tuned\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.218715 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.218526    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:18:00.218715 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.218595    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:18:00.218715 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.218611    2583 projected.go:196] Error preparing data for projected volume kube-api-access-mvh42 for pod openshift-network-diagnostics/network-check-target-tgn5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:00.218715 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.218683    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42 podName:8eab1d96-88bb-417a-b03e-8e2f0274e862 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:00.718665319 +0000 UTC m=+3.105933145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mvh42" (UniqueName: "kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42") pod "network-check-target-tgn5k" (UID: "8eab1d96-88bb-417a-b03e-8e2f0274e862") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:00.226642 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.226622    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sfkt\" (UniqueName: \"kubernetes.io/projected/407cf470-1b82-424c-a8e2-a5f061b3b84c-kube-api-access-7sfkt\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:00.228073 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.228023    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm75h\" (UniqueName: \"kubernetes.io/projected/87c8cdc5-be2e-47a0-8cde-e944fda51056-kube-api-access-wm75h\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.228175 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.228132    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5gx\" (UniqueName: \"kubernetes.io/projected/92a4974a-3e6e-4575-b5bd-303beb3fa201-kube-api-access-jp5gx\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:00.228708 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.228685    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgv2b\" (UniqueName: \"kubernetes.io/projected/012d43c7-e3f2-4931-877e-cdf66eaf61b6-kube-api-access-sgv2b\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.230667 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.230629    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzxg\" (UniqueName: \"kubernetes.io/projected/428997c4-b34e-4e3c-8551-c2d8510c69f4-kube-api-access-kqzxg\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:00.295978 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.295771    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:00.308800 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:00.307674    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3bd8549_385d_4ad6_8fac_cd2e7b4b37e6.slice/crio-46263b706eaf4cfcc2d5ae28dacaa4a4ccd35a80ec5b8274f7f82a9e42799b0b WatchSource:0}: Error finding container 46263b706eaf4cfcc2d5ae28dacaa4a4ccd35a80ec5b8274f7f82a9e42799b0b: Status 404 returned error can't find the container with id 46263b706eaf4cfcc2d5ae28dacaa4a4ccd35a80ec5b8274f7f82a9e42799b0b
Oct 30 22:18:00.318725 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.318702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:00.328349 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.328325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:00.328861 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:00.328831    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40f9c33_c4fe_4176_9ae9_561592a770f6.slice/crio-0509d0cf50d58ba7f871bd87079bee6a384612bcae5b2ed7c44494859ee66310 WatchSource:0}: Error finding container 0509d0cf50d58ba7f871bd87079bee6a384612bcae5b2ed7c44494859ee66310: Status 404 returned error can't find the container with id 0509d0cf50d58ba7f871bd87079bee6a384612bcae5b2ed7c44494859ee66310
Oct 30 22:18:00.334574 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.334549    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:00.336948 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.336803    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:00.341714 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.341694    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:00.344104 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.344083    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kfshl"
Oct 30 22:18:00.345324 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:00.345180    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60457f9c_65b0_4547_9012_2fc4efe2b9e8.slice/crio-6afb2c7f1be5f304d205eeb403dad888f0f128ab0260e09b4e54435c893ee416 WatchSource:0}: Error finding container 6afb2c7f1be5f304d205eeb403dad888f0f128ab0260e09b4e54435c893ee416: Status 404 returned error can't find the container with id 6afb2c7f1be5f304d205eeb403dad888f0f128ab0260e09b4e54435c893ee416
Oct 30 22:18:00.349172 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.348976    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:00.351252 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.351156    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:00.351760 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:00.351733    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a4974a_3e6e_4575_b5bd_303beb3fa201.slice/crio-ec63bb01333f4ac171a9de591aa0030b6a7f80fd57913288b79fe49c64bbc597 WatchSource:0}: Error finding container ec63bb01333f4ac171a9de591aa0030b6a7f80fd57913288b79fe49c64bbc597: Status 404 returned error can't find the container with id ec63bb01333f4ac171a9de591aa0030b6a7f80fd57913288b79fe49c64bbc597
Oct 30 22:18:00.354657 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:00.354636    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0170e7_99d6_4102_9b88_13552cb6f421.slice/crio-afa2b2cc445d9210c603695d7342906088b6a9ab69fe0776876ca02ce9505ffc WatchSource:0}: Error finding container afa2b2cc445d9210c603695d7342906088b6a9ab69fe0776876ca02ce9505ffc: Status 404 returned error can't find the container with id afa2b2cc445d9210c603695d7342906088b6a9ab69fe0776876ca02ce9505ffc
Oct 30 22:18:00.370623 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:00.370591    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012d43c7_e3f2_4931_877e_cdf66eaf61b6.slice/crio-3b50d991919905d92c08b2984b4a815d24f1dcaa5af4416a730b76b6b79dd2fe WatchSource:0}: Error finding container 3b50d991919905d92c08b2984b4a815d24f1dcaa5af4416a730b76b6b79dd2fe: Status 404 returned error can't find the container with id 3b50d991919905d92c08b2984b4a815d24f1dcaa5af4416a730b76b6b79dd2fe
Oct 30 22:18:00.371653 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:00.371628    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af08ecc_3199_4c8b_b364_f42f4befeff1.slice/crio-0e69d09c541b7f90c5b73e2effd2c7846cf2d1059e71f811d091108f27cbfe01 WatchSource:0}: Error finding container 0e69d09c541b7f90c5b73e2effd2c7846cf2d1059e71f811d091108f27cbfe01: Status 404 returned error can't find the container with id 0e69d09c541b7f90c5b73e2effd2c7846cf2d1059e71f811d091108f27cbfe01
Oct 30 22:18:00.709280 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.709197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:00.709280 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.709250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:00.709502 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.709375    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:00.709502 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.709404    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:00.709502 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.709442    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret podName:adb5dbb9-f122-43fc-9ef9-867e6b2bb700 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:01.709425172 +0000 UTC m=+4.096693022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret") pod "global-pull-secret-syncer-xkt2l" (UID: "adb5dbb9-f122-43fc-9ef9-867e6b2bb700") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:00.709502 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.709460    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs podName:428997c4-b34e-4e3c-8551-c2d8510c69f4 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:01.709454107 +0000 UTC m=+4.096721934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs") pod "network-metrics-daemon-hmgsj" (UID: "428997c4-b34e-4e3c-8551-c2d8510c69f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:00.810054 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:00.810011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:00.821514 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.818800    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:18:00.821514 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.818831    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:18:00.821514 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.818850    2583 projected.go:196] Error preparing data for projected volume kube-api-access-mvh42 for pod openshift-network-diagnostics/network-check-target-tgn5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:00.821514 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:00.818910    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42 podName:8eab1d96-88bb-417a-b03e-8e2f0274e862 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:01.818889065 +0000 UTC m=+4.206156906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvh42" (UniqueName: "kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42") pod "network-check-target-tgn5k" (UID: "8eab1d96-88bb-417a-b03e-8e2f0274e862") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:01.052103 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.051903    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:13:00 +0000 UTC" deadline="2028-06-19 17:32:01.716328822 +0000 UTC"
Oct 30 22:18:01.052103 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.051984    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14347h14m0.664358755s"
Oct 30 22:18:01.065187 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.064518    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:01.065187 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:01.064675    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:01.101087 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.100983    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wb9bv" event={"ID":"60457f9c-65b0-4547-9012-2fc4efe2b9e8","Type":"ContainerStarted","Data":"6afb2c7f1be5f304d205eeb403dad888f0f128ab0260e09b4e54435c893ee416"}
Oct 30 22:18:01.108272 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.108236    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64cvm" event={"ID":"b40f9c33-c4fe-4176-9ae9-561592a770f6","Type":"ContainerStarted","Data":"0509d0cf50d58ba7f871bd87079bee6a384612bcae5b2ed7c44494859ee66310"}
Oct 30 22:18:01.137189 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.137148    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ggnd6" event={"ID":"012d43c7-e3f2-4931-877e-cdf66eaf61b6","Type":"ContainerStarted","Data":"3b50d991919905d92c08b2984b4a815d24f1dcaa5af4416a730b76b6b79dd2fe"}
Oct 30 22:18:01.142048 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.141872    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff" event={"ID":"9af08ecc-3199-4c8b-b364-f42f4befeff1","Type":"ContainerStarted","Data":"0e69d09c541b7f90c5b73e2effd2c7846cf2d1059e71f811d091108f27cbfe01"}
Oct 30 22:18:01.158385 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.158318    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kfshl" event={"ID":"87c8cdc5-be2e-47a0-8cde-e944fda51056","Type":"ContainerStarted","Data":"83f8805eb5ebe70e2d9ebecf1c77dd7eb1aae9c735e184ff97480b2e56d9ad22"}
Oct 30 22:18:01.176397 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.176304    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl" event={"ID":"de0170e7-99d6-4102-9b88-13552cb6f421","Type":"ContainerStarted","Data":"afa2b2cc445d9210c603695d7342906088b6a9ab69fe0776876ca02ce9505ffc"}
Oct 30 22:18:01.195401 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.195368    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7cjdl" event={"ID":"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6","Type":"ContainerStarted","Data":"46263b706eaf4cfcc2d5ae28dacaa4a4ccd35a80ec5b8274f7f82a9e42799b0b"}
Oct 30 22:18:01.201342 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.201303    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vhkmj" event={"ID":"407cf470-1b82-424c-a8e2-a5f061b3b84c","Type":"ContainerStarted","Data":"2ea75aed4060b21a6e5b4e68c6bbd8c4c0f466d2268d1bf40d2c983630c341be"}
Oct 30 22:18:01.214818 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.214757    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qqwss" event={"ID":"92a4974a-3e6e-4575-b5bd-303beb3fa201","Type":"ContainerStarted","Data":"ec63bb01333f4ac171a9de591aa0030b6a7f80fd57913288b79fe49c64bbc597"}
Oct 30 22:18:01.716671 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.716627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:01.716861 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.716685    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:01.716942 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:01.716926    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:01.717015 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:01.717001    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs podName:428997c4-b34e-4e3c-8551-c2d8510c69f4 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:03.716980577 +0000 UTC m=+6.104248422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs") pod "network-metrics-daemon-hmgsj" (UID: "428997c4-b34e-4e3c-8551-c2d8510c69f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:01.717788 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:01.717765    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:01.717906 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:01.717857    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret podName:adb5dbb9-f122-43fc-9ef9-867e6b2bb700 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:03.71781252 +0000 UTC m=+6.105080360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret") pod "global-pull-secret-syncer-xkt2l" (UID: "adb5dbb9-f122-43fc-9ef9-867e6b2bb700") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:01.919431 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:01.919290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:01.919617 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:01.919431    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:18:01.919617 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:01.919447    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:18:01.919617 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:01.919457    2583 projected.go:196] Error preparing data for projected volume kube-api-access-mvh42 for pod openshift-network-diagnostics/network-check-target-tgn5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:01.919617 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:01.919531    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42 podName:8eab1d96-88bb-417a-b03e-8e2f0274e862 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:03.919516651 +0000 UTC m=+6.306784494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvh42" (UniqueName: "kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42") pod "network-check-target-tgn5k" (UID: "8eab1d96-88bb-417a-b03e-8e2f0274e862") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:02.053019 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:02.052931    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:13:00 +0000 UTC" deadline="2028-06-15 21:24:35.832898691 +0000 UTC"
Oct 30 22:18:02.053019 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:02.052972    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14255h6m33.779931581s"
Oct 30 22:18:02.072641 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:02.067635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:02.072641 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:02.067793    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:02.072641 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:02.068204    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:02.072641 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:02.068305    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:03.064654 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:03.064375    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:03.065229 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:03.064776    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:03.244039 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:03.243907    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" event={"ID":"a5c727ac326546386befce51c4d99c6d","Type":"ContainerStarted","Data":"605458c066ca7a377d2619a0af91912e10f5c6825d4bb0948428c733d5cdb681"}
Oct 30 22:18:03.263563 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:03.263529    2583 generic.go:356] "Generic (PLEG): container finished" podID="65deeb7675a6280ac03893337db51c32" containerID="3642d671d5430b07486cc8c49fec1a1474b3d96262c1fd6903f545a48a5e8cea" exitCode=0
Oct 30 22:18:03.263760 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:03.263596    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal" event={"ID":"65deeb7675a6280ac03893337db51c32","Type":"ContainerDied","Data":"3642d671d5430b07486cc8c49fec1a1474b3d96262c1fd6903f545a48a5e8cea"}
Oct 30 22:18:03.265047 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:03.265000    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" podStartSLOduration=4.264988619 podStartE2EDuration="4.264988619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:03.264413172 +0000 UTC m=+5.651681023" watchObservedRunningTime="2026-10-30 22:18:03.264988619 +0000 UTC m=+5.652256469"
Oct 30 22:18:03.735122 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:03.734280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:03.735122 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:03.734337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:03.735122 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:03.734505    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:03.735122 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:03.734588    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs podName:428997c4-b34e-4e3c-8551-c2d8510c69f4 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:07.734568197 +0000 UTC m=+10.121836048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs") pod "network-metrics-daemon-hmgsj" (UID: "428997c4-b34e-4e3c-8551-c2d8510c69f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:03.735122 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:03.735010    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:03.735122 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:03.735059    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret podName:adb5dbb9-f122-43fc-9ef9-867e6b2bb700 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:07.735043314 +0000 UTC m=+10.122311157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret") pod "global-pull-secret-syncer-xkt2l" (UID: "adb5dbb9-f122-43fc-9ef9-867e6b2bb700") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:03.936070 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:03.935424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:03.936070 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:03.935656    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:18:03.936070 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:03.935671    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:18:03.936070 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:03.935681    2583 projected.go:196] Error preparing data for projected volume kube-api-access-mvh42 for pod openshift-network-diagnostics/network-check-target-tgn5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:03.936070 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:03.935724    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42 podName:8eab1d96-88bb-417a-b03e-8e2f0274e862 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:07.935712433 +0000 UTC m=+10.322980271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvh42" (UniqueName: "kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42") pod "network-check-target-tgn5k" (UID: "8eab1d96-88bb-417a-b03e-8e2f0274e862") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:04.067181 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:04.067149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:04.067662 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:04.067323    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:04.067771 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:04.067753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:04.067886 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:04.067863    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:04.273996 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:04.273912    2583 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="4c34b2608bfaf95f34f80ee411671799b36815c4699aa7384fcccce776a80ccf" exitCode=0
Oct 30 22:18:04.274155 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:04.274006    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64cvm" event={"ID":"b40f9c33-c4fe-4176-9ae9-561592a770f6","Type":"ContainerDied","Data":"4c34b2608bfaf95f34f80ee411671799b36815c4699aa7384fcccce776a80ccf"}
Oct 30 22:18:04.282880 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:04.282825    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal" event={"ID":"65deeb7675a6280ac03893337db51c32","Type":"ContainerStarted","Data":"049607b4cfec0b036c6396af3a97baccd7224a0379d2a97d0be157456f3c6717"}
Oct 30 22:18:05.064351 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:05.064279    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:05.064527 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:05.064389    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:05.289336 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:05.289302    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wb9bv" event={"ID":"60457f9c-65b0-4547-9012-2fc4efe2b9e8","Type":"ContainerStarted","Data":"4750f7aa32ffe5108abbcca3257cf4be29d83532221d3eb8c97469a47d8aa1cc"}
Oct 30 22:18:05.308676 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:05.308618    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal" podStartSLOduration=6.308596178 podStartE2EDuration="6.308596178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:04.321608203 +0000 UTC m=+6.708876052" watchObservedRunningTime="2026-10-30 22:18:05.308596178 +0000 UTC m=+7.695864027"
Oct 30 22:18:06.067657 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:06.064377    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:06.067657 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:06.064545    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:06.067657 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:06.067454    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:06.067657 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:06.067569    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:06.300644 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:06.300527    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7cjdl" event={"ID":"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6","Type":"ContainerStarted","Data":"e0406c20bd15c5c221acd4b23f2608c2bfa82308a856aa964e71ae37b059420a"}
Oct 30 22:18:06.318582 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:06.317268    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wb9bv" podStartSLOduration=4.096398873 podStartE2EDuration="8.317250661s" totalImagesPullingTime="4.220851788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:05.309590703 +0000 UTC m=+7.696858551" watchObservedRunningTime="2026-10-30 22:18:06.317250661 +0000 UTC m=+8.704518508"
Oct 30 22:18:07.024249 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:07.024174    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:07.024772 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:07.024751    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:07.042473 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:07.042422    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7cjdl" podStartSLOduration=3.640779995 podStartE2EDuration="9.042374001s" totalImagesPullingTime="5.401594006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:06.318526332 +0000 UTC m=+8.705794180" watchObservedRunningTime="2026-10-30 22:18:07.042374001 +0000 UTC m=+9.429641845"
Oct 30 22:18:07.064661 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:07.064630    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:07.064830 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:07.064759    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:07.305653 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:07.305576    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl" event={"ID":"de0170e7-99d6-4102-9b88-13552cb6f421","Type":"ContainerStarted","Data":"6d53fdffaf88e3c226a8988672a4ab037c61179e2ec48be88d2f62306dd5e3b4"}
Oct 30 22:18:07.771876 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:07.771803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:07.771876 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:07.771860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:07.772044 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:07.771993    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:07.772079 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:07.772056    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs podName:428997c4-b34e-4e3c-8551-c2d8510c69f4 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:15.772038099 +0000 UTC m=+18.159305939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs") pod "network-metrics-daemon-hmgsj" (UID: "428997c4-b34e-4e3c-8551-c2d8510c69f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:07.772468 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:07.772446    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:07.772583 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:07.772520    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret podName:adb5dbb9-f122-43fc-9ef9-867e6b2bb700 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:15.772488801 +0000 UTC m=+18.159756631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret") pod "global-pull-secret-syncer-xkt2l" (UID: "adb5dbb9-f122-43fc-9ef9-867e6b2bb700") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:07.972966 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:07.972932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:07.973182 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:07.973135    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:18:07.973182 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:07.973165    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:18:07.973182 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:07.973177    2583 projected.go:196] Error preparing data for projected volume kube-api-access-mvh42 for pod openshift-network-diagnostics/network-check-target-tgn5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:07.973371 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:07.973230    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42 podName:8eab1d96-88bb-417a-b03e-8e2f0274e862 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:15.973211853 +0000 UTC m=+18.360479684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvh42" (UniqueName: "kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42") pod "network-check-target-tgn5k" (UID: "8eab1d96-88bb-417a-b03e-8e2f0274e862") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:08.067739 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:08.067704    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:08.067884 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:08.067840    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:08.068282 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:08.068258    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:08.068383 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:08.068358    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:08.146963 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:08.146835    2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:18:08.198839 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:08.198113    2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:18:08.146857059Z","UUID":"79c5874c-8054-468e-accc-6e8951e5400e","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:18:08.200548 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:08.200523    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:18:08.200548 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:08.200555    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:18:08.312842 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:08.312770    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl" event={"ID":"de0170e7-99d6-4102-9b88-13552cb6f421","Type":"ContainerStarted","Data":"a74961a2c413e7ded6598e421a77a356328bc0dce1ad87e09a7b7b1881ac69ea"}
Oct 30 22:18:08.314726 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:08.314702    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:18:08.315620 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:08.315598    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kfshl" event={"ID":"87c8cdc5-be2e-47a0-8cde-e944fda51056","Type":"ContainerStarted","Data":"6d04a8a9df3812f244163bc177495baec7ebf17501300e9addc916ca31c83e22"}
Oct 30 22:18:09.065148 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:09.065115    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:09.065328 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:09.065239    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:09.318130 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:09.318086    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qqwss" event={"ID":"92a4974a-3e6e-4575-b5bd-303beb3fa201","Type":"ContainerStarted","Data":"c2f2c7dbaa4f4e404d41c8edd01a698f73f8f1e23a91d6095cfaf9646cea1bf5"}
Oct 30 22:18:09.320670 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:09.320646    2583 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="c8500d159b7b748246b129391cb66578b407c042f241c861acfa2a9dff08da6a" exitCode=0
Oct 30 22:18:09.320793 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:09.320711    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64cvm" event={"ID":"b40f9c33-c4fe-4176-9ae9-561592a770f6","Type":"ContainerDied","Data":"c8500d159b7b748246b129391cb66578b407c042f241c861acfa2a9dff08da6a"}
Oct 30 22:18:09.346343 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:09.346275    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qqwss" podStartSLOduration=2.9785407189999997 podStartE2EDuration="11.34625867s" totalImagesPullingTime="8.367717951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:09.346182415 +0000 UTC m=+11.733450264" watchObservedRunningTime="2026-10-30 22:18:09.34625867 +0000 UTC m=+11.733526519"
Oct 30 22:18:09.346957 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:09.346916    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kfshl" podStartSLOduration=2.501160087 podStartE2EDuration="10.346907124s" totalImagesPullingTime="7.845747037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:08.340827093 +0000 UTC m=+10.728094943" watchObservedRunningTime="2026-10-30 22:18:09.346907124 +0000 UTC m=+11.734174972"
Oct 30 22:18:10.064224 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:10.064194    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:10.064367 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:10.064332    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:10.064826 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:10.064803    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:10.064936 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:10.064919    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:10.326056 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:10.325983    2583 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="018f861db55ca7375d1476023c7c1437a079cd134a5df91a9e67f3f678ab0e92" exitCode=0
Oct 30 22:18:10.326560 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:10.326089    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64cvm" event={"ID":"b40f9c33-c4fe-4176-9ae9-561592a770f6","Type":"ContainerDied","Data":"018f861db55ca7375d1476023c7c1437a079cd134a5df91a9e67f3f678ab0e92"}
Oct 30 22:18:10.328990 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:10.328962    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl" event={"ID":"de0170e7-99d6-4102-9b88-13552cb6f421","Type":"ContainerStarted","Data":"d90036658b3e47c667c8c41825be5099bcab022b64d45655bb7f3ad7a6bdaf99"}
Oct 30 22:18:10.331359 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:10.331041    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vhkmj" event={"ID":"407cf470-1b82-424c-a8e2-a5f061b3b84c","Type":"ContainerStarted","Data":"bbb16423fd51eb0e68d3d1e3dfa67143bede8466650c175b8b38032a6264199c"}
Oct 30 22:18:10.378094 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:10.378039    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl" podStartSLOduration=3.909856856 podStartE2EDuration="12.378026655s" totalImagesPullingTime="8.468169799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:10.377845194 +0000 UTC m=+12.765113043" watchObservedRunningTime="2026-10-30 22:18:10.378026655 +0000 UTC m=+12.765294501"
Oct 30 22:18:10.394880 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:10.394827    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vhkmj" podStartSLOduration=4.22313007 podStartE2EDuration="12.394814569s" totalImagesPullingTime="8.171684499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:10.394383366 +0000 UTC m=+12.781651213" watchObservedRunningTime="2026-10-30 22:18:10.394814569 +0000 UTC m=+12.782082414"
Oct 30 22:18:11.064607 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:11.064381    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:11.064730 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:11.064691    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:11.336246 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:11.336212    2583 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="4c62e21dfdb661764b385623f9297abcc452aafed2dcf7fe10400cee140fffb3" exitCode=0
Oct 30 22:18:11.336777 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:11.336237    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64cvm" event={"ID":"b40f9c33-c4fe-4176-9ae9-561592a770f6","Type":"ContainerDied","Data":"4c62e21dfdb661764b385623f9297abcc452aafed2dcf7fe10400cee140fffb3"}
Oct 30 22:18:11.338106 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:11.338077    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ggnd6" event={"ID":"012d43c7-e3f2-4931-877e-cdf66eaf61b6","Type":"ContainerStarted","Data":"2bb0140915b058a8a667de674f09e931bc2ca561d4c6ae7f9a607e92f61116cb"}
Oct 30 22:18:11.381615 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:11.381566    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ggnd6" podStartSLOduration=1.75317409 podStartE2EDuration="12.381555655s" totalImagesPullingTime="10.628381565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:11.381493222 +0000 UTC m=+13.768761070" watchObservedRunningTime="2026-10-30 22:18:11.381555655 +0000 UTC m=+13.768823499"
Oct 30 22:18:12.065221 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:12.065186    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:12.065221 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:12.065211    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:12.065428 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:12.065315    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:12.065793 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:12.065770    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:13.064926 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:13.064894    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:13.065630 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:13.065016    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:14.064966 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:14.064934    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:14.065597 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:14.065043    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:14.065597 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:14.065366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:14.065597 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:14.065471    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:14.347776 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:14.347690    2583 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="d508648ad38bb794d6872a25e1424eea0727b1b88134e91d31ecb23617ec0b2d" exitCode=0
Oct 30 22:18:14.347776 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:14.347748    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64cvm" event={"ID":"b40f9c33-c4fe-4176-9ae9-561592a770f6","Type":"ContainerDied","Data":"d508648ad38bb794d6872a25e1424eea0727b1b88134e91d31ecb23617ec0b2d"}
Oct 30 22:18:15.064494 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:15.064463    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:15.064679 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:15.064566    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:15.352435 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:15.352351    2583 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="81b5c6a914092fc16adfff1186008ec03f8e7b9d64448acc5a6b0555ff903d48" exitCode=0
Oct 30 22:18:15.352435 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:15.352392    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64cvm" event={"ID":"b40f9c33-c4fe-4176-9ae9-561592a770f6","Type":"ContainerDied","Data":"81b5c6a914092fc16adfff1186008ec03f8e7b9d64448acc5a6b0555ff903d48"}
Oct 30 22:18:15.831357 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:15.831146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:15.831492 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:15.831369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:15.831492 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:15.831312    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:15.831492 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:15.831484    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:15.831633 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:15.831495    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret podName:adb5dbb9-f122-43fc-9ef9-867e6b2bb700 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:31.831475531 +0000 UTC m=+34.218743379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret") pod "global-pull-secret-syncer-xkt2l" (UID: "adb5dbb9-f122-43fc-9ef9-867e6b2bb700") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:18:15.831633 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:15.831552    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs podName:428997c4-b34e-4e3c-8551-c2d8510c69f4 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:31.831536605 +0000 UTC m=+34.218804433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs") pod "network-metrics-daemon-hmgsj" (UID: "428997c4-b34e-4e3c-8551-c2d8510c69f4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:18:16.033275 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.033243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:16.033442 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:16.033393    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:18:16.033442 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:16.033410    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:18:16.033442 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:16.033419    2583 projected.go:196] Error preparing data for projected volume kube-api-access-mvh42 for pod openshift-network-diagnostics/network-check-target-tgn5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:16.033561 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:16.033465    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42 podName:8eab1d96-88bb-417a-b03e-8e2f0274e862 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:32.033452383 +0000 UTC m=+34.420720207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvh42" (UniqueName: "kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42") pod "network-check-target-tgn5k" (UID: "8eab1d96-88bb-417a-b03e-8e2f0274e862") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:18:16.065216 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.065193    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:16.065356 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:16.065333    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:16.065403 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.065379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:16.065469 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:16.065453    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:16.357797 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.357714    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-64cvm" event={"ID":"b40f9c33-c4fe-4176-9ae9-561592a770f6","Type":"ContainerStarted","Data":"aa55f97452662661d0ac48a7c6d1a853e816e20475b6c432d70934f980b7cbd5"}
Oct 30 22:18:16.360562 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.360537    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff" event={"ID":"9af08ecc-3199-4c8b-b364-f42f4befeff1","Type":"ContainerStarted","Data":"7eb2772801ac6dde462531485c605e53885b746426ef76c1a7afba54ef3bba08"}
Oct 30 22:18:16.360684 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.360612    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff" event={"ID":"9af08ecc-3199-4c8b-b364-f42f4befeff1","Type":"ContainerStarted","Data":"f89a099d42fed3697e418f11479bc32cd8c7e3beafa5defbb76cb37a6628a13a"}
Oct 30 22:18:16.360684 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.360657    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff" event={"ID":"9af08ecc-3199-4c8b-b364-f42f4befeff1","Type":"ContainerStarted","Data":"d7865109d1ac224dd1439e73822fb32937f50c0d457d83d4639ea4faa4c81963"}
Oct 30 22:18:16.360805 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.360700    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff" event={"ID":"9af08ecc-3199-4c8b-b364-f42f4befeff1","Type":"ContainerStarted","Data":"cc312b1514b4054ffe22fcd8097bcd95afd1b444b8b2ee86f30d04e97aa28c2b"}
Oct 30 22:18:16.360805 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.360747    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff" event={"ID":"9af08ecc-3199-4c8b-b364-f42f4befeff1","Type":"ContainerStarted","Data":"f88ef6ff1c2da2c7e5bee05d93128077a85c87422330604451b70f09252cde3b"}
Oct 30 22:18:16.360805 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.360793    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff" event={"ID":"9af08ecc-3199-4c8b-b364-f42f4befeff1","Type":"ContainerStarted","Data":"0a3a6ee272a461298d53c553e2e49e787762028ad23101418f6f2cd13bf91b71"}
Oct 30 22:18:16.384674 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:16.384623    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-64cvm" podStartSLOduration=7.444881113 podStartE2EDuration="18.384611559s" totalImagesPullingTime="10.939730446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:16.383194676 +0000 UTC m=+18.770462501" watchObservedRunningTime="2026-10-30 22:18:16.384611559 +0000 UTC m=+18.771879405"
Oct 30 22:18:17.064580 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:17.064390    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:17.064757 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:17.064648    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:18.065133 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:18.065058    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:18.065537 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:18.065149    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:18.065537 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:18.065236    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:18.065537 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:18.065346    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:18.367660 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:18.367431    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff" event={"ID":"9af08ecc-3199-4c8b-b364-f42f4befeff1","Type":"ContainerStarted","Data":"f729af0c1948ce0cd50d274ca72c9e3687171b10c3951be24e9c86126c251f93"}
Oct 30 22:18:18.993485 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:18.993451    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:18.993679 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:18.993616    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:18:18.994017 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:18.993992    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:19.064103 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:19.064079    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:19.064235 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:19.064175    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:20.064613 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:20.064579    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:20.064613 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:20.064602    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:20.065403 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:20.064693    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:20.065403 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:20.064818    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:20.375313 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:20.375280    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff" event={"ID":"9af08ecc-3199-4c8b-b364-f42f4befeff1","Type":"ContainerStarted","Data":"c2e0eab85ad56f6fd17391e0f2eb6f6e101d61309451a5c49d7d1ee5fcf44bdb"}
Oct 30 22:18:20.375551 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:20.375533    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:20.390444 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:20.390423    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:20.409691 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:20.409658    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff" podStartSLOduration=7.357003886 podStartE2EDuration="22.409645952s" totalImagesPullingTime="15.052642066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:20.409341108 +0000 UTC m=+22.796608954" watchObservedRunningTime="2026-10-30 22:18:20.409645952 +0000 UTC m=+22.796913798"
Oct 30 22:18:21.065168 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:21.065131    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:21.065559 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:21.065242    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:21.377526 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:21.377441    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:21.377647 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:21.377531    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:21.391792 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:21.391767    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:21.610147 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:21.610118    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xkt2l"]
Oct 30 22:18:21.610280 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:21.610242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:21.610348 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:21.610330    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:21.611705 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:21.611682    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgn5k"]
Oct 30 22:18:21.611797 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:21.611764    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:21.611850 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:21.611833    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:21.614875 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:21.614854    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hmgsj"]
Oct 30 22:18:21.614960 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:21.614948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:21.615057 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:21.615041    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:23.064644 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:23.064616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:23.065152 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:23.064732    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:23.065152 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:23.064791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:23.065152 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:23.064877    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:24.064972 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:24.064937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:24.065363 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:24.065056    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:25.064478 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:25.064289    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:25.064647 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:25.064317    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:25.064647 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:25.064566    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:25.064716 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:25.064663    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:26.064797 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:26.064723    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:26.065204 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:26.064831    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hmgsj" podUID="428997c4-b34e-4e3c-8551-c2d8510c69f4"
Oct 30 22:18:27.064245 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.064207    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:27.064245 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.064240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:27.064462 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:27.064319    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tgn5k" podUID="8eab1d96-88bb-417a-b03e-8e2f0274e862"
Oct 30 22:18:27.064462 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:27.064438    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xkt2l" podUID="adb5dbb9-f122-43fc-9ef9-867e6b2bb700"
Oct 30 22:18:27.839998 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.839971    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeReady"
Oct 30 22:18:27.840354 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.840081    2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 22:18:27.878069 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.878002    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tsv8h"]
Oct 30 22:18:27.880237 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.880221    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tsv8h"
Oct 30 22:18:27.881322 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.881300    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b4mzj"]
Oct 30 22:18:27.883908 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.883890    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8cbpq"]
Oct 30 22:18:27.886863 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.884240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b4mzj"
Oct 30 22:18:27.888691 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.888667    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xkcch"]
Oct 30 22:18:27.890827 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.890453    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.890827 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.890573    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.890939 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.890914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8cbpq"
Oct 30 22:18:27.892390 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.892370    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8"]
Oct 30 22:18:27.893143 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.892613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:27.895138 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.895120    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"]
Oct 30 22:18:27.896269 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.896240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8"
Oct 30 22:18:27.897911 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.897891    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"]
Oct 30 22:18:27.898779 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.898764    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:27.901302 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.901285    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6668ff59b7-vglrn"]
Oct 30 22:18:27.902111 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.902089    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:27.906108 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.906090    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"]
Oct 30 22:18:27.907178 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.906346    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:27.908789 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.908769    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 22:18:27.910710 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.910398    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 22:18:27.910767 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.910748    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-grpzl"]
Oct 30 22:18:27.910857 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.910841    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:27.913787 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.913769    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"]
Oct 30 22:18:27.913946 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.913926    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:27.916856 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.916837    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"]
Oct 30 22:18:27.916955 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.916941    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:27.922537 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.922484    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bfb9545c9-tkcrm"]
Oct 30 22:18:27.925025 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.923298    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:27.927159 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.927142    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6"]
Oct 30 22:18:27.927417 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.927403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:27.934638 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.934613    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5559797b78-pnvl9"]
Oct 30 22:18:27.935756 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.935736    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6"
Oct 30 22:18:27.941297 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.941277    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tsv8h"]
Oct 30 22:18:27.941395 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.941306    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8cbpq"]
Oct 30 22:18:27.941395 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.941329    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rvz8k"]
Oct 30 22:18:27.941395 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.941375    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:27.945279 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.945261    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6wrrt"]
Oct 30 22:18:27.945376 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.945362    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:27.948086 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948069    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8"]
Oct 30 22:18:27.948192 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948095    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xkcch"]
Oct 30 22:18:27.948192 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948109    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"]
Oct 30 22:18:27.948192 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948122    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"]
Oct 30 22:18:27.948192 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948136    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"]
Oct 30 22:18:27.948192 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948153    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b4mzj"]
Oct 30 22:18:27.948192 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948171    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6668ff59b7-vglrn"]
Oct 30 22:18:27.948192 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948184    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"]
Oct 30 22:18:27.948624 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948199    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-grpzl"]
Oct 30 22:18:27.948624 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948206    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:27.948624 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948210    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6wrrt"]
Oct 30 22:18:27.948624 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948367    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6"]
Oct 30 22:18:27.948624 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948395    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rvz8k"]
Oct 30 22:18:27.948624 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948416    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"]
Oct 30 22:18:27.948624 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948438    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5559797b78-pnvl9"]
Oct 30 22:18:27.948624 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.948457    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bfb9545c9-tkcrm"]
Oct 30 22:18:27.988896 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.988874    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.989019 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.988988    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.989087 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989028    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.989087 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989056    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 22:18:27.989087 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989085    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.989254 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989098    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 22:18:27.989254 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989145    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.989254 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989181    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.989254 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989213    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.989254 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989238    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.989558 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989263    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 22:18:27.989558 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989333    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 22:18:27.989558 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989344    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.989558 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989418    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:18:27.989558 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989444    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.989558 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989484    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.989870 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989857    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.989950 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.989861    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.990565 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.990441    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.990565 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.990443    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.990565 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.990466    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 22:18:27.990565 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.990467    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.990565 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.990540    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.990779 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.990577    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 22:18:27.991050 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.991031    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.991157 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.991076    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:18:27.991157 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.991102    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:18:27.991602 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.991584    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:18:27.995628 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:27.995606    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 22:18:28.002009 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.001987    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 22:18:28.002309 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.002290    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 22:18:28.002420 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.002406    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:18:28.002707 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.002681    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 22:18:28.003283 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003260    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 22:18:28.003357 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003273    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 22:18:28.003357 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003341    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 22:18:28.003470 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003386    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 22:18:28.003470 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003415    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 22:18:28.003470 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003448    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 22:18:28.003470 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003450    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 22:18:28.003673 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003478    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 22:18:28.003673 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003501    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:18:28.003673 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003542    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 22:18:28.003673 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003608    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9jqsq\""
Oct 30 22:18:28.003874 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003672    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 22:18:28.003874 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003705    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-p44cw\""
Oct 30 22:18:28.003874 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003731    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-snj5d\""
Oct 30 22:18:28.003874 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003757    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jfcbv\""
Oct 30 22:18:28.003874 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.003805    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:18:28.014770 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.014745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:28.014853 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.014781    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-trusted-ca\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.014853 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.014832    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-default-certificate\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.014973 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.014903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.014973 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.014947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vmt\" (UniqueName: \"kubernetes.io/projected/188a085b-a512-440b-90ec-30acc66e4171-kube-api-access-86vmt\") pod \"ingress-canary-rvz8k\" (UID: \"188a085b-a512-440b-90ec-30acc66e4171\") " pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:28.015048 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.014977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.015048 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-installation-pull-secrets\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.015133 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tndn\" (UniqueName: \"kubernetes.io/projected/1889bdc0-508e-4f97-80cb-4a5aa6e4f67f-kube-api-access-9tndn\") pod \"kas-connection-checker-685895c7c9-tsv8h\" (UID: \"1889bdc0-508e-4f97-80cb-4a5aa6e4f67f\") " pod="kube-system/kas-connection-checker-685895c7c9-tsv8h"
Oct 30 22:18:28.015133 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8199465-5b50-41fd-ab4f-5201977736a3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:28.015219 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:28.015219 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:28.015313 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq55l\" (UniqueName: \"kubernetes.io/projected/3a7d925c-af25-4c40-a280-e870b59cb772-kube-api-access-gq55l\") pod \"volume-data-source-validator-58c9f5f46d-tw8c8\" (UID: \"3a7d925c-af25-4c40-a280-e870b59cb772\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8"
Oct 30 22:18:28.015313 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015295    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f16a8be-4c89-4786-a385-ff738661c9bb-serving-cert\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:28.015377 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e71c29d-cd72-42ae-a0de-1500b78c89b0-config-volume\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.015377 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-trusted-ca\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.015377 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-service-ca-bundle\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.015528 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015478    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.015591 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqfzc\" (UniqueName: \"kubernetes.io/projected/baf3e5ab-01d9-4caa-a239-b6df965e21e4-kube-api-access-nqfzc\") pod \"kas-connection-checker-685895c7c9-b4mzj\" (UID: \"baf3e5ab-01d9-4caa-a239-b6df965e21e4\") " pod="kube-system/kas-connection-checker-685895c7c9-b4mzj"
Oct 30 22:18:28.015644 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-certificates\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.015644 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8m5m\" (UniqueName: \"kubernetes.io/projected/2e346445-18a9-494c-9b3f-6c0860f631f7-kube-api-access-t8m5m\") pod \"kas-connection-checker-685895c7c9-8cbpq\" (UID: \"2e346445-18a9-494c-9b3f-6c0860f631f7\") " pod="kube-system/kas-connection-checker-685895c7c9-8cbpq"
Oct 30 22:18:28.015710 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.015710 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq7tw\" (UniqueName: \"kubernetes.io/projected/16ece3f9-76fd-4c93-b915-d1c8cec664fb-kube-api-access-nq7tw\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.015781 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:28.015781 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88d59947-036f-412d-9ee6-f7565b0d4d8b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:28.015781 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92gt\" (UniqueName: \"kubernetes.io/projected/88c3c753-4382-452d-898d-732766a65bf1-kube-api-access-j92gt\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.015914 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2x9v\" (UniqueName: \"kubernetes.io/projected/53de1b04-dea0-4a6f-9609-760792cc0406-kube-api-access-f2x9v\") pod \"network-check-source-6f78d59f99-5s5g6\" (UID: \"53de1b04-dea0-4a6f-9609-760792cc0406\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6"
Oct 30 22:18:28.015914 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88c3c753-4382-452d-898d-732766a65bf1-serving-cert\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.015914 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62crk\" (UniqueName: \"kubernetes.io/projected/9f16a8be-4c89-4786-a385-ff738661c9bb-kube-api-access-62crk\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:28.016002 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xd26\" (UniqueName: \"kubernetes.io/projected/8e71c29d-cd72-42ae-a0de-1500b78c89b0-kube-api-access-9xd26\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.016002 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.015942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16ece3f9-76fd-4c93-b915-d1c8cec664fb-serving-cert\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.016059 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvzt\" (UniqueName: \"kubernetes.io/projected/99252037-fc1d-4055-b66b-2f0e8ab910b8-kube-api-access-dlvzt\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.016090 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-installation-pull-secrets\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.016118 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-snapshots\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.016168 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-bound-sa-token\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.016168 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-tmp\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.016267 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-registry-certificates\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.016267 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-config\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.016267 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e71c29d-cd72-42ae-a0de-1500b78c89b0-tmp-dir\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.016390 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:28.016390 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-image-registry-private-configuration\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.016390 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-image-registry-private-configuration\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.016542 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016404    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v87rd\" (UniqueName: \"kubernetes.io/projected/83a06c56-145a-4a52-baca-2d01c679e581-kube-api-access-v87rd\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:28.016542 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-trusted-ca\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.016542 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-kube-api-access-rjxjz\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.016669 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7kc\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-kube-api-access-xl7kc\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.016669 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/188a085b-a512-440b-90ec-30acc66e4171-cert\") pod \"ingress-canary-rvz8k\" (UID: \"188a085b-a512-440b-90ec-30acc66e4171\") " pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:28.016669 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-ca-trust-extracted\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.016669 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.016828 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2vtz\" (UniqueName: \"kubernetes.io/projected/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-kube-api-access-p2vtz\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:28.016828 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8199465-5b50-41fd-ab4f-5201977736a3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:28.016828 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f16a8be-4c89-4786-a385-ff738661c9bb-config\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:28.016828 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-bound-sa-token\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.016947 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/555a97b6-857c-4374-8939-1eed2d487b4c-ca-trust-extracted\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.016947 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-stats-auth\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.017014 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.017014 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.016972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8kds\" (UniqueName: \"kubernetes.io/projected/f8199465-5b50-41fd-ab4f-5201977736a3-kube-api-access-v8kds\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:28.064417 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.064382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:28.103446 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.103415    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:18:28.103651 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.103632    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ms5nl\""
Oct 30 22:18:28.117447 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-config\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.117447 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e71c29d-cd72-42ae-a0de-1500b78c89b0-tmp-dir\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.117652 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:28.117652 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-image-registry-private-configuration\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.117652 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-image-registry-private-configuration\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.117652 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v87rd\" (UniqueName: \"kubernetes.io/projected/83a06c56-145a-4a52-baca-2d01c679e581-kube-api-access-v87rd\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:28.117652 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-trusted-ca\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.117652 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-kube-api-access-rjxjz\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7kc\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-kube-api-access-xl7kc\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/188a085b-a512-440b-90ec-30acc66e4171-cert\") pod \"ingress-canary-rvz8k\" (UID: \"188a085b-a512-440b-90ec-30acc66e4171\") " pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-ca-trust-extracted\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2vtz\" (UniqueName: \"kubernetes.io/projected/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-kube-api-access-p2vtz\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8199465-5b50-41fd-ab4f-5201977736a3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f16a8be-4c89-4786-a385-ff738661c9bb-config\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-bound-sa-token\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/555a97b6-857c-4374-8939-1eed2d487b4c-ca-trust-extracted\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-stats-auth\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117911    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e71c29d-cd72-42ae-a0de-1500b78c89b0-tmp-dir\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.117934 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kds\" (UniqueName: \"kubernetes.io/projected/f8199465-5b50-41fd-ab4f-5201977736a3-kube-api-access-v8kds\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.117975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-trusted-ca\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-default-certificate\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vmt\" (UniqueName: \"kubernetes.io/projected/188a085b-a512-440b-90ec-30acc66e4171-kube-api-access-86vmt\") pod \"ingress-canary-rvz8k\" (UID: \"188a085b-a512-440b-90ec-30acc66e4171\") " pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-installation-pull-secrets\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tndn\" (UniqueName: \"kubernetes.io/projected/1889bdc0-508e-4f97-80cb-4a5aa6e4f67f-kube-api-access-9tndn\") pod \"kas-connection-checker-685895c7c9-tsv8h\" (UID: \"1889bdc0-508e-4f97-80cb-4a5aa6e4f67f\") " pod="kube-system/kas-connection-checker-685895c7c9-tsv8h"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.118169    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118242    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-config\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.118260    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert podName:88d59947-036f-412d-9ee6-f7565b0d4d8b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:28.618241002 +0000 UTC m=+31.005508840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-grpzl" (UID: "88d59947-036f-412d-9ee6-f7565b0d4d8b") : secret "networking-console-plugin-cert" not found
Oct 30 22:18:28.118393 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118172    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-ca-trust-extracted\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8199465-5b50-41fd-ab4f-5201977736a3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118438    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f16a8be-4c89-4786-a385-ff738661c9bb-config\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq55l\" (UniqueName: \"kubernetes.io/projected/3a7d925c-af25-4c40-a280-e870b59cb772-kube-api-access-gq55l\") pod \"volume-data-source-validator-58c9f5f46d-tw8c8\" (UID: \"3a7d925c-af25-4c40-a280-e870b59cb772\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f16a8be-4c89-4786-a385-ff738661c9bb-serving-cert\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e71c29d-cd72-42ae-a0de-1500b78c89b0-config-volume\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-trusted-ca\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-service-ca-bundle\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqfzc\" (UniqueName: \"kubernetes.io/projected/baf3e5ab-01d9-4caa-a239-b6df965e21e4-kube-api-access-nqfzc\") pod \"kas-connection-checker-685895c7c9-b4mzj\" (UID: \"baf3e5ab-01d9-4caa-a239-b6df965e21e4\") " pod="kube-system/kas-connection-checker-685895c7c9-b4mzj"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-certificates\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8m5m\" (UniqueName: \"kubernetes.io/projected/2e346445-18a9-494c-9b3f-6c0860f631f7-kube-api-access-t8m5m\") pod \"kas-connection-checker-685895c7c9-8cbpq\" (UID: \"2e346445-18a9-494c-9b3f-6c0860f631f7\") " pod="kube-system/kas-connection-checker-685895c7c9-8cbpq"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.118913    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:18:28.119085 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq7tw\" (UniqueName: \"kubernetes.io/projected/16ece3f9-76fd-4c93-b915-d1c8cec664fb-kube-api-access-nq7tw\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.118968    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls podName:83a06c56-145a-4a52-baca-2d01c679e581 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:28.618954601 +0000 UTC m=+31.006222453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r6fws" (UID: "83a06c56-145a-4a52-baca-2d01c679e581") : secret "samples-operator-tls" not found
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.118987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88d59947-036f-412d-9ee6-f7565b0d4d8b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92gt\" (UniqueName: \"kubernetes.io/projected/88c3c753-4382-452d-898d-732766a65bf1-kube-api-access-j92gt\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119051    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-trusted-ca\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2x9v\" (UniqueName: \"kubernetes.io/projected/53de1b04-dea0-4a6f-9609-760792cc0406-kube-api-access-f2x9v\") pod \"network-check-source-6f78d59f99-5s5g6\" (UID: \"53de1b04-dea0-4a6f-9609-760792cc0406\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88c3c753-4382-452d-898d-732766a65bf1-serving-cert\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62crk\" (UniqueName: \"kubernetes.io/projected/9f16a8be-4c89-4786-a385-ff738661c9bb-kube-api-access-62crk\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xd26\" (UniqueName: \"kubernetes.io/projected/8e71c29d-cd72-42ae-a0de-1500b78c89b0-kube-api-access-9xd26\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119147    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16ece3f9-76fd-4c93-b915-d1c8cec664fb-serving-cert\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvzt\" (UniqueName: \"kubernetes.io/projected/99252037-fc1d-4055-b66b-2f0e8ab910b8-kube-api-access-dlvzt\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-installation-pull-secrets\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-snapshots\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.119883 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-bound-sa-token\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.120643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-tmp\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.120643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-registry-certificates\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.120643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119414    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/555a97b6-857c-4374-8939-1eed2d487b4c-ca-trust-extracted\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.120643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.119944    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-registry-certificates\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.120643 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.120273    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle podName:99252037-fc1d-4055-b66b-2f0e8ab910b8 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:28.620256581 +0000 UTC m=+31.007524430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle") pod "router-default-6668ff59b7-vglrn" (UID: "99252037-fc1d-4055-b66b-2f0e8ab910b8") : configmap references non-existent config key: service-ca.crt
Oct 30 22:18:28.120643 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.120313    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:18:28.120643 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.120374    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs podName:99252037-fc1d-4055-b66b-2f0e8ab910b8 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:28.620358051 +0000 UTC m=+31.007625901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs") pod "router-default-6668ff59b7-vglrn" (UID: "99252037-fc1d-4055-b66b-2f0e8ab910b8") : secret "router-metrics-certs-default" not found
Oct 30 22:18:28.120643 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.120573    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:28.121028 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.120748    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:18:28.121028 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.120765    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfb9545c9-tkcrm: secret "image-registry-tls" not found
Oct 30 22:18:28.121028 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.120805    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls podName:2d4504af-560b-4e5a-a3cf-befff8d8d6d2 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:28.620791035 +0000 UTC m=+31.008058863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls") pod "image-registry-5bfb9545c9-tkcrm" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2") : secret "image-registry-tls" not found
Oct 30 22:18:28.121028 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.120871    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:28.121028 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.120909    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls podName:c69ae355-a045-48d1-b7c9-881d2c4e8a1b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:28.620897451 +0000 UTC m=+31.008165277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4vs7k" (UID: "c69ae355-a045-48d1-b7c9-881d2c4e8a1b") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:28.121929 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.121388    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8199465-5b50-41fd-ab4f-5201977736a3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:28.121929 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.121710    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88d59947-036f-412d-9ee6-f7565b0d4d8b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:28.122437 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.122412    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:28.123062 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.123035    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-tmp\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.123151 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.123103    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-snapshots\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.123428 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.123407    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-trusted-ca\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.123621 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.123540    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-image-registry-private-configuration\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.123724 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.123625    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f16a8be-4c89-4786-a385-ff738661c9bb-serving-cert\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:28.124005 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.123980    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:18:28.124005 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.124005    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5559797b78-pnvl9: secret "image-registry-tls" not found
Oct 30 22:18:28.124150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.124046    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls podName:555a97b6-857c-4374-8939-1eed2d487b4c nodeName:}" failed. No retries permitted until 2026-10-30 22:18:28.624032268 +0000 UTC m=+31.011300112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls") pod "image-registry-5559797b78-pnvl9" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c") : secret "image-registry-tls" not found
Oct 30 22:18:28.124348 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.124327    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-service-ca-bundle\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.125025 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.124932    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:18:28.125025 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.124986    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls podName:8e71c29d-cd72-42ae-a0de-1500b78c89b0 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:28.62497093 +0000 UTC m=+31.012238771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls") pod "dns-default-6wrrt" (UID: "8e71c29d-cd72-42ae-a0de-1500b78c89b0") : secret "dns-default-metrics-tls" not found
Oct 30 22:18:28.125169 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.125053    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e71c29d-cd72-42ae-a0de-1500b78c89b0-config-volume\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.125169 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.125089    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-trusted-ca\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.125364 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.125283    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-certificates\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.126981 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.126739    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-stats-auth\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.126981 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.126887    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-installation-pull-secrets\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.126981 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.126940    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-default-certificate\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.127336 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.127043    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88c3c753-4382-452d-898d-732766a65bf1-serving-cert\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.127336 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.127095    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-installation-pull-secrets\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.127336 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.127246    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8199465-5b50-41fd-ab4f-5201977736a3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:28.127485 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.127441    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/188a085b-a512-440b-90ec-30acc66e4171-cert\") pod \"ingress-canary-rvz8k\" (UID: \"188a085b-a512-440b-90ec-30acc66e4171\") " pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:28.127937 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.127903    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16ece3f9-76fd-4c93-b915-d1c8cec664fb-serving-cert\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.128909 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.128860    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-image-registry-private-configuration\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.132312 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.132292    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v87rd\" (UniqueName: \"kubernetes.io/projected/83a06c56-145a-4a52-baca-2d01c679e581-kube-api-access-v87rd\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:28.133577 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.133554    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq7tw\" (UniqueName: \"kubernetes.io/projected/16ece3f9-76fd-4c93-b915-d1c8cec664fb-kube-api-access-nq7tw\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.134482 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.134450    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2vtz\" (UniqueName: \"kubernetes.io/projected/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-kube-api-access-p2vtz\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:28.134726 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.134702    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-bound-sa-token\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.135101 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.135079    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xd26\" (UniqueName: \"kubernetes.io/projected/8e71c29d-cd72-42ae-a0de-1500b78c89b0-kube-api-access-9xd26\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.135611 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.135592    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7kc\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-kube-api-access-xl7kc\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.136041 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.136024    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq55l\" (UniqueName: \"kubernetes.io/projected/3a7d925c-af25-4c40-a280-e870b59cb772-kube-api-access-gq55l\") pod \"volume-data-source-validator-58c9f5f46d-tw8c8\" (UID: \"3a7d925c-af25-4c40-a280-e870b59cb772\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8"
Oct 30 22:18:28.136629 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.136608    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92gt\" (UniqueName: \"kubernetes.io/projected/88c3c753-4382-452d-898d-732766a65bf1-kube-api-access-j92gt\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.137860 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.137838    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-kube-api-access-rjxjz\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.138552 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.138492    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tndn\" (UniqueName: \"kubernetes.io/projected/1889bdc0-508e-4f97-80cb-4a5aa6e4f67f-kube-api-access-9tndn\") pod \"kas-connection-checker-685895c7c9-tsv8h\" (UID: \"1889bdc0-508e-4f97-80cb-4a5aa6e4f67f\") " pod="kube-system/kas-connection-checker-685895c7c9-tsv8h"
Oct 30 22:18:28.138983 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.138965    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vmt\" (UniqueName: \"kubernetes.io/projected/188a085b-a512-440b-90ec-30acc66e4171-kube-api-access-86vmt\") pod \"ingress-canary-rvz8k\" (UID: \"188a085b-a512-440b-90ec-30acc66e4171\") " pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:28.139272 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.139250    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-bound-sa-token\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.139938 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.139910    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2x9v\" (UniqueName: \"kubernetes.io/projected/53de1b04-dea0-4a6f-9609-760792cc0406-kube-api-access-f2x9v\") pod \"network-check-source-6f78d59f99-5s5g6\" (UID: \"53de1b04-dea0-4a6f-9609-760792cc0406\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6"
Oct 30 22:18:28.140140 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.140105    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kds\" (UniqueName: \"kubernetes.io/projected/f8199465-5b50-41fd-ab4f-5201977736a3-kube-api-access-v8kds\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:28.140769 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.140749    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvzt\" (UniqueName: \"kubernetes.io/projected/99252037-fc1d-4055-b66b-2f0e8ab910b8-kube-api-access-dlvzt\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.140840 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.140758    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8m5m\" (UniqueName: \"kubernetes.io/projected/2e346445-18a9-494c-9b3f-6c0860f631f7-kube-api-access-t8m5m\") pod \"kas-connection-checker-685895c7c9-8cbpq\" (UID: \"2e346445-18a9-494c-9b3f-6c0860f631f7\") " pod="kube-system/kas-connection-checker-685895c7c9-8cbpq"
Oct 30 22:18:28.141895 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.141872    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqfzc\" (UniqueName: \"kubernetes.io/projected/baf3e5ab-01d9-4caa-a239-b6df965e21e4-kube-api-access-nqfzc\") pod \"kas-connection-checker-685895c7c9-b4mzj\" (UID: \"baf3e5ab-01d9-4caa-a239-b6df965e21e4\") " pod="kube-system/kas-connection-checker-685895c7c9-b4mzj"
Oct 30 22:18:28.141975 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.141899    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62crk\" (UniqueName: \"kubernetes.io/projected/9f16a8be-4c89-4786-a385-ff738661c9bb-kube-api-access-62crk\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:28.197072 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.197051    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tsv8h"
Oct 30 22:18:28.208915 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.208893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b4mzj"
Oct 30 22:18:28.218256 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.218009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8cbpq"
Oct 30 22:18:28.226858 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.226836    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:28.246708 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.243756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8"
Oct 30 22:18:28.267359 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.267330    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:28.279346 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.279169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:28.284776 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.284745    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:28.304245 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.303867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6"
Oct 30 22:18:28.325770 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.324697    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:28.497226 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.497198    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tsv8h"]
Oct 30 22:18:28.502890 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.502155    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b4mzj"]
Oct 30 22:18:28.502890 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:28.502633    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1889bdc0_508e_4f97_80cb_4a5aa6e4f67f.slice/crio-e55f947e1675bf5ae5340da6e81b272c1df6274582811d79d0f8defc375aad86 WatchSource:0}: Error finding container e55f947e1675bf5ae5340da6e81b272c1df6274582811d79d0f8defc375aad86: Status 404 returned error can't find the container with id e55f947e1675bf5ae5340da6e81b272c1df6274582811d79d0f8defc375aad86
Oct 30 22:18:28.504994 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:28.504901    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf3e5ab_01d9_4caa_a239_b6df965e21e4.slice/crio-49a89c7c4427f3075d63943fc81dd663470b547b2471beb6f7981c7bab57c811 WatchSource:0}: Error finding container 49a89c7c4427f3075d63943fc81dd663470b547b2471beb6f7981c7bab57c811: Status 404 returned error can't find the container with id 49a89c7c4427f3075d63943fc81dd663470b547b2471beb6f7981c7bab57c811
Oct 30 22:18:28.512268 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.512245    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8cbpq"]
Oct 30 22:18:28.518838 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:28.518816    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e346445_18a9_494c_9b3f_6c0860f631f7.slice/crio-fc3cef33e0cca3abe20249fbbd2709a37c6793194d800f29a59f1e5955859074 WatchSource:0}: Error finding container fc3cef33e0cca3abe20249fbbd2709a37c6793194d800f29a59f1e5955859074: Status 404 returned error can't find the container with id fc3cef33e0cca3abe20249fbbd2709a37c6793194d800f29a59f1e5955859074
Oct 30 22:18:28.560947 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.560913    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8"]
Oct 30 22:18:28.574499 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:28.574472    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7d925c_af25_4c40_a280_e870b59cb772.slice/crio-458275610f6e084d5e480f1ef033cbdb6418b1d08834cd383adff7c4982a0715 WatchSource:0}: Error finding container 458275610f6e084d5e480f1ef033cbdb6418b1d08834cd383adff7c4982a0715: Status 404 returned error can't find the container with id 458275610f6e084d5e480f1ef033cbdb6418b1d08834cd383adff7c4982a0715
Oct 30 22:18:28.615872 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.615840    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6"]
Oct 30 22:18:28.618305 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.618279    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"]
Oct 30 22:18:28.618762 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:28.618736    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53de1b04_dea0_4a6f_9609_760792cc0406.slice/crio-53d20e4b5c3853d4a42880651f1ccd76d8168056eefce8b175a655fcbb8d6323 WatchSource:0}: Error finding container 53d20e4b5c3853d4a42880651f1ccd76d8168056eefce8b175a655fcbb8d6323: Status 404 returned error can't find the container with id 53d20e4b5c3853d4a42880651f1ccd76d8168056eefce8b175a655fcbb8d6323
Oct 30 22:18:28.619268 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.619067    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"]
Oct 30 22:18:28.621760 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:28.621738    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8199465_5b50_41fd_ab4f_5201977736a3.slice/crio-00a6f191b21baf3cff1ae517e24ae8f98dad92eda7e8c5aa5e3d1a12f5c1179e WatchSource:0}: Error finding container 00a6f191b21baf3cff1ae517e24ae8f98dad92eda7e8c5aa5e3d1a12f5c1179e: Status 404 returned error can't find the container with id 00a6f191b21baf3cff1ae517e24ae8f98dad92eda7e8c5aa5e3d1a12f5c1179e
Oct 30 22:18:28.622675 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:28.622616    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c3c753_4382_452d_898d_732766a65bf1.slice/crio-ac29696864f0f5cbe7d0bd530b05d9acbe4351b370c863f7a47b0cd3e2eefc15 WatchSource:0}: Error finding container ac29696864f0f5cbe7d0bd530b05d9acbe4351b370c863f7a47b0cd3e2eefc15: Status 404 returned error can't find the container with id ac29696864f0f5cbe7d0bd530b05d9acbe4351b370c863f7a47b0cd3e2eefc15
Oct 30 22:18:28.626491 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.626450    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xkcch"]
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.629481    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.629537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.629570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.629599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.629631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.629677    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.629708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.629831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.629938    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.629974    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert podName:88d59947-036f-412d-9ee6-f7565b0d4d8b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:29.62996352 +0000 UTC m=+32.017231345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-grpzl" (UID: "88d59947-036f-412d-9ee6-f7565b0d4d8b") : secret "networking-console-plugin-cert" not found
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630016    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630048    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs podName:99252037-fc1d-4055-b66b-2f0e8ab910b8 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:29.630041409 +0000 UTC m=+32.017309233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs") pod "router-default-6668ff59b7-vglrn" (UID: "99252037-fc1d-4055-b66b-2f0e8ab910b8") : secret "router-metrics-certs-default" not found
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630088    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:28.630725 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630119    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls podName:c69ae355-a045-48d1-b7c9-881d2c4e8a1b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:29.630112373 +0000 UTC m=+32.017380196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4vs7k" (UID: "c69ae355-a045-48d1-b7c9-881d2c4e8a1b") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630172    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle podName:99252037-fc1d-4055-b66b-2f0e8ab910b8 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:29.630165005 +0000 UTC m=+32.017432830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle") pod "router-default-6668ff59b7-vglrn" (UID: "99252037-fc1d-4055-b66b-2f0e8ab910b8") : configmap references non-existent config key: service-ca.crt
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630228    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630239    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfb9545c9-tkcrm: secret "image-registry-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630280    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls podName:2d4504af-560b-4e5a-a3cf-befff8d8d6d2 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:29.630270402 +0000 UTC m=+32.017538231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls") pod "image-registry-5bfb9545c9-tkcrm" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2") : secret "image-registry-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630330    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630356    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls podName:83a06c56-145a-4a52-baca-2d01c679e581 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:29.63034746 +0000 UTC m=+32.017615284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r6fws" (UID: "83a06c56-145a-4a52-baca-2d01c679e581") : secret "samples-operator-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630400    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630408    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5559797b78-pnvl9: secret "image-registry-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630431    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls podName:555a97b6-857c-4374-8939-1eed2d487b4c nodeName:}" failed. No retries permitted until 2026-10-30 22:18:29.630424106 +0000 UTC m=+32.017691932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls") pod "image-registry-5559797b78-pnvl9" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c") : secret "image-registry-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630476    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:28.630499    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls podName:8e71c29d-cd72-42ae-a0de-1500b78c89b0 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:29.630490332 +0000 UTC m=+32.017758159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls") pod "dns-default-6wrrt" (UID: "8e71c29d-cd72-42ae-a0de-1500b78c89b0") : secret "dns-default-metrics-tls" not found
Oct 30 22:18:28.635150 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.634886    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"]
Oct 30 22:18:28.640729 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:28.640703    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f16a8be_4c89_4786_a385_ff738661c9bb.slice/crio-064263fb24d1f1530e87ccd92388446309aac371f65d39927ceffeadc0e43185 WatchSource:0}: Error finding container 064263fb24d1f1530e87ccd92388446309aac371f65d39927ceffeadc0e43185: Status 404 returned error can't find the container with id 064263fb24d1f1530e87ccd92388446309aac371f65d39927ceffeadc0e43185
Oct 30 22:18:28.654324 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:28.654301    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rvz8k"]
Oct 30 22:18:29.064478 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.064434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:29.064478 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.064462    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:29.070769 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.070748    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-r4dw4\""
Oct 30 22:18:29.070929 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.070896    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:18:29.070929 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.070902    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sdfwl\""
Oct 30 22:18:29.412560 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.412400    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rvz8k" event={"ID":"188a085b-a512-440b-90ec-30acc66e4171","Type":"ContainerStarted","Data":"24f6e5a3ac4313c52e40175843de822b11bd947e711492b5fb031d95fab873fb"}
Oct 30 22:18:29.416034 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.415974    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6" event={"ID":"53de1b04-dea0-4a6f-9609-760792cc0406","Type":"ContainerStarted","Data":"53d20e4b5c3853d4a42880651f1ccd76d8168056eefce8b175a655fcbb8d6323"}
Oct 30 22:18:29.426105 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.426074    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h" event={"ID":"9f16a8be-4c89-4786-a385-ff738661c9bb","Type":"ContainerStarted","Data":"064263fb24d1f1530e87ccd92388446309aac371f65d39927ceffeadc0e43185"}
Oct 30 22:18:29.428559 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.428268    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tsv8h" event={"ID":"1889bdc0-508e-4f97-80cb-4a5aa6e4f67f","Type":"ContainerStarted","Data":"7be38c521aa446fef06db3c2fe1ba5b7470c13bc2683b9c4844974c484b3d2ab"}
Oct 30 22:18:29.428559 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.428341    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tsv8h" event={"ID":"1889bdc0-508e-4f97-80cb-4a5aa6e4f67f","Type":"ContainerStarted","Data":"e55f947e1675bf5ae5340da6e81b272c1df6274582811d79d0f8defc375aad86"}
Oct 30 22:18:29.430245 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.430181    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8" event={"ID":"3a7d925c-af25-4c40-a280-e870b59cb772","Type":"ContainerStarted","Data":"458275610f6e084d5e480f1ef033cbdb6418b1d08834cd383adff7c4982a0715"}
Oct 30 22:18:29.432802 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.432722    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xkcch" event={"ID":"16ece3f9-76fd-4c93-b915-d1c8cec664fb","Type":"ContainerStarted","Data":"e39b79b239aba08b4d306d91ee3e02b66c2352419b90dba87222d6a154cc462d"}
Oct 30 22:18:29.436723 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.436182    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8cbpq" event={"ID":"2e346445-18a9-494c-9b3f-6c0860f631f7","Type":"ContainerStarted","Data":"d9176a77f32f528bf46c6b17136f82c5c80f1cbe2134ca3fc21dd01a5fd65e76"}
Oct 30 22:18:29.436723 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.436225    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8cbpq" event={"ID":"2e346445-18a9-494c-9b3f-6c0860f631f7","Type":"ContainerStarted","Data":"fc3cef33e0cca3abe20249fbbd2709a37c6793194d800f29a59f1e5955859074"}
Oct 30 22:18:29.441772 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.441747    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b4mzj" event={"ID":"baf3e5ab-01d9-4caa-a239-b6df965e21e4","Type":"ContainerStarted","Data":"ea7b64fa35efeae24b1eaa54dbc2f36318fe9b931a41c94ba5b2a78f578ea324"}
Oct 30 22:18:29.441871 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.441791    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b4mzj" event={"ID":"baf3e5ab-01d9-4caa-a239-b6df965e21e4","Type":"ContainerStarted","Data":"49a89c7c4427f3075d63943fc81dd663470b547b2471beb6f7981c7bab57c811"}
Oct 30 22:18:29.444195 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.444172    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" event={"ID":"88c3c753-4382-452d-898d-732766a65bf1","Type":"ContainerStarted","Data":"ac29696864f0f5cbe7d0bd530b05d9acbe4351b370c863f7a47b0cd3e2eefc15"}
Oct 30 22:18:29.448325 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.448278    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tsv8h" podStartSLOduration=336.448262635 podStartE2EDuration="5m36.448262635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:29.44667293 +0000 UTC m=+31.833940780" watchObservedRunningTime="2026-10-30 22:18:29.448262635 +0000 UTC m=+31.835530483"
Oct 30 22:18:29.449203 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.449180    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b" event={"ID":"f8199465-5b50-41fd-ab4f-5201977736a3","Type":"ContainerStarted","Data":"00a6f191b21baf3cff1ae517e24ae8f98dad92eda7e8c5aa5e3d1a12f5c1179e"}
Oct 30 22:18:29.465689 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.465539    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-b4mzj" podStartSLOduration=336.465523739 podStartE2EDuration="5m36.465523739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:29.465055187 +0000 UTC m=+31.852323034" watchObservedRunningTime="2026-10-30 22:18:29.465523739 +0000 UTC m=+31.852791585"
Oct 30 22:18:29.483901 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.483856    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8cbpq" podStartSLOduration=336.483840676 podStartE2EDuration="5m36.483840676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:29.48326559 +0000 UTC m=+31.870533437" watchObservedRunningTime="2026-10-30 22:18:29.483840676 +0000 UTC m=+31.871108526"
Oct 30 22:18:29.640609 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.640572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:29.640760 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.640626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:29.640760 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.640652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:29.640760 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.640675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:29.640760 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.640706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:29.640760 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.640752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:29.641005 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.640794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:29.641005 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:29.640885    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:29.641091 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641056    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:18:29.641133 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641119    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert podName:88d59947-036f-412d-9ee6-f7565b0d4d8b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:31.64110114 +0000 UTC m=+34.028368984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-grpzl" (UID: "88d59947-036f-412d-9ee6-f7565b0d4d8b") : secret "networking-console-plugin-cert" not found
Oct 30 22:18:29.641554 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641527    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:18:29.641668 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641580    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs podName:99252037-fc1d-4055-b66b-2f0e8ab910b8 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:31.64156632 +0000 UTC m=+34.028834149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs") pod "router-default-6668ff59b7-vglrn" (UID: "99252037-fc1d-4055-b66b-2f0e8ab910b8") : secret "router-metrics-certs-default" not found
Oct 30 22:18:29.641668 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641641    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:29.641783 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641678    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls podName:c69ae355-a045-48d1-b7c9-881d2c4e8a1b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:31.641664134 +0000 UTC m=+34.028931958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4vs7k" (UID: "c69ae355-a045-48d1-b7c9-881d2c4e8a1b") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:29.641783 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641743    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle podName:99252037-fc1d-4055-b66b-2f0e8ab910b8 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:31.641733061 +0000 UTC m=+34.029000890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle") pod "router-default-6668ff59b7-vglrn" (UID: "99252037-fc1d-4055-b66b-2f0e8ab910b8") : configmap references non-existent config key: service-ca.crt
Oct 30 22:18:29.641888 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641804    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:18:29.641888 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641815    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfb9545c9-tkcrm: secret "image-registry-tls" not found
Oct 30 22:18:29.641888 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641844    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls podName:2d4504af-560b-4e5a-a3cf-befff8d8d6d2 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:31.641834703 +0000 UTC m=+34.029102529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls") pod "image-registry-5bfb9545c9-tkcrm" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2") : secret "image-registry-tls" not found
Oct 30 22:18:29.642028 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641899    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:18:29.642028 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641933    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls podName:83a06c56-145a-4a52-baca-2d01c679e581 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:31.641922182 +0000 UTC m=+34.029190009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r6fws" (UID: "83a06c56-145a-4a52-baca-2d01c679e581") : secret "samples-operator-tls" not found
Oct 30 22:18:29.642028 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641988    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:18:29.642028 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.641998    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5559797b78-pnvl9: secret "image-registry-tls" not found
Oct 30 22:18:29.642215 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.642029    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls podName:555a97b6-857c-4374-8939-1eed2d487b4c nodeName:}" failed. No retries permitted until 2026-10-30 22:18:31.642018651 +0000 UTC m=+34.029286491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls") pod "image-registry-5559797b78-pnvl9" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c") : secret "image-registry-tls" not found
Oct 30 22:18:29.642215 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.642085    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:18:29.642215 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:29.642115    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls podName:8e71c29d-cd72-42ae-a0de-1500b78c89b0 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:31.642105111 +0000 UTC m=+34.029372940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls") pod "dns-default-6wrrt" (UID: "8e71c29d-cd72-42ae-a0de-1500b78c89b0") : secret "dns-default-metrics-tls" not found
Oct 30 22:18:31.464717 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.463903    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8" event={"ID":"3a7d925c-af25-4c40-a280-e870b59cb772","Type":"ContainerStarted","Data":"54000333c851729e073d2d6dc459774081543c4b0a796aa925ff6cc4e0253cb3"}
Oct 30 22:18:31.664795 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.664713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:31.664795 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.664771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:31.665109 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.664807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:31.665109 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.664862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:31.665109 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.664928    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle podName:99252037-fc1d-4055-b66b-2f0e8ab910b8 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:35.6649056 +0000 UTC m=+38.052173442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle") pod "router-default-6668ff59b7-vglrn" (UID: "99252037-fc1d-4055-b66b-2f0e8ab910b8") : configmap references non-existent config key: service-ca.crt
Oct 30 22:18:31.665109 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.664973    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:18:31.665109 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.664977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:31.665109 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.664989    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5559797b78-pnvl9: secret "image-registry-tls" not found
Oct 30 22:18:31.665109 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.665080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665150    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls podName:555a97b6-857c-4374-8939-1eed2d487b4c nodeName:}" failed. No retries permitted until 2026-10-30 22:18:35.665127446 +0000 UTC m=+38.052395287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls") pod "image-registry-5559797b78-pnvl9" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c") : secret "image-registry-tls" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.664973    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665189    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665200    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls podName:83a06c56-145a-4a52-baca-2d01c679e581 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:35.665189156 +0000 UTC m=+38.052457019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r6fws" (UID: "83a06c56-145a-4a52-baca-2d01c679e581") : secret "samples-operator-tls" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.665228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665235    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert podName:88d59947-036f-412d-9ee6-f7565b0d4d8b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:35.665220671 +0000 UTC m=+38.052488499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-grpzl" (UID: "88d59947-036f-412d-9ee6-f7565b0d4d8b") : secret "networking-console-plugin-cert" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665036    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665263    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfb9545c9-tkcrm: secret "image-registry-tls" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.665268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665289    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665048    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665314    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls podName:2d4504af-560b-4e5a-a3cf-befff8d8d6d2 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:35.665297752 +0000 UTC m=+38.052565579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls") pod "image-registry-5bfb9545c9-tkcrm" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2") : secret "image-registry-tls" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665348    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs podName:99252037-fc1d-4055-b66b-2f0e8ab910b8 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:35.665336473 +0000 UTC m=+38.052604301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs") pod "router-default-6668ff59b7-vglrn" (UID: "99252037-fc1d-4055-b66b-2f0e8ab910b8") : secret "router-metrics-certs-default" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665362    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls podName:8e71c29d-cd72-42ae-a0de-1500b78c89b0 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:35.665353565 +0000 UTC m=+38.052621394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls") pod "dns-default-6wrrt" (UID: "8e71c29d-cd72-42ae-a0de-1500b78c89b0") : secret "dns-default-metrics-tls" not found
Oct 30 22:18:31.665454 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665382    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:31.666063 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.665427    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls podName:c69ae355-a045-48d1-b7c9-881d2c4e8a1b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:35.665416115 +0000 UTC m=+38.052683944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4vs7k" (UID: "c69ae355-a045-48d1-b7c9-881d2c4e8a1b") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:31.867519 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.867461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:31.867711 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.867617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:31.868520 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.867769    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:18:31.868520 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:31.867839    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs podName:428997c4-b34e-4e3c-8551-c2d8510c69f4 nodeName:}" failed. No retries permitted until 2026-10-30 22:19:03.86782126 +0000 UTC m=+66.255089089 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs") pod "network-metrics-daemon-hmgsj" (UID: "428997c4-b34e-4e3c-8551-c2d8510c69f4") : secret "metrics-daemon-secret" not found
Oct 30 22:18:31.870875 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:31.870851    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:32.069526 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:32.069486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:32.073817 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:32.073789    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:32.105434 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:32.105398    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:32.108890 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:32.108861    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:32.426056 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:32.424295    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8" podStartSLOduration=312.368566655 podStartE2EDuration="5m14.424275251s" totalImagesPullingTime="2.055708596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:31.484637767 +0000 UTC m=+33.871905613" watchObservedRunningTime="2026-10-30 22:18:32.424275251 +0000 UTC m=+34.811543102"
Oct 30 22:18:32.426056 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:32.425939    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tgn5k"]
Oct 30 22:18:32.427496 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:32.427467    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eab1d96_88bb_417a_b03e_8e2f0274e862.slice/crio-46523a1fca3e6d49c6fd38149fad1450dded49837a8cd1edadaf4ab6ebbd07b8 WatchSource:0}: Error finding container 46523a1fca3e6d49c6fd38149fad1450dded49837a8cd1edadaf4ab6ebbd07b8: Status 404 returned error can't find the container with id 46523a1fca3e6d49c6fd38149fad1450dded49837a8cd1edadaf4ab6ebbd07b8
Oct 30 22:18:32.465241 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:32.465153    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xkt2l"]
Oct 30 22:18:32.468126 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:32.468096    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb5dbb9_f122_43fc_9ef9_867e6b2bb700.slice/crio-92b9a4460b7b1f4186ae630e84c9a29174ad749c12ff3b9640d144dfde700fc6 WatchSource:0}: Error finding container 92b9a4460b7b1f4186ae630e84c9a29174ad749c12ff3b9640d144dfde700fc6: Status 404 returned error can't find the container with id 92b9a4460b7b1f4186ae630e84c9a29174ad749c12ff3b9640d144dfde700fc6
Oct 30 22:18:32.473630 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:32.473586    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgn5k" event={"ID":"8eab1d96-88bb-417a-b03e-8e2f0274e862","Type":"ContainerStarted","Data":"46523a1fca3e6d49c6fd38149fad1450dded49837a8cd1edadaf4ab6ebbd07b8"}
Oct 30 22:18:33.478808 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.478762    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xkt2l" event={"ID":"adb5dbb9-f122-43fc-9ef9-867e6b2bb700","Type":"ContainerStarted","Data":"92b9a4460b7b1f4186ae630e84c9a29174ad749c12ff3b9640d144dfde700fc6"}
Oct 30 22:18:33.483033 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.482140    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/0.log"
Oct 30 22:18:33.483033 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.482175    2583 generic.go:356] "Generic (PLEG): container finished" podID="88c3c753-4382-452d-898d-732766a65bf1" containerID="370f815774aa68d2abc729082052137182679357706cf311925c92831d1b9a11" exitCode=255
Oct 30 22:18:33.483033 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.482281    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" event={"ID":"88c3c753-4382-452d-898d-732766a65bf1","Type":"ContainerDied","Data":"370f815774aa68d2abc729082052137182679357706cf311925c92831d1b9a11"}
Oct 30 22:18:33.483033 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.482587    2583 scope.go:122] "RemoveContainer" containerID="370f815774aa68d2abc729082052137182679357706cf311925c92831d1b9a11"
Oct 30 22:18:33.485392 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.485263    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xkcch" event={"ID":"16ece3f9-76fd-4c93-b915-d1c8cec664fb","Type":"ContainerStarted","Data":"25770bf587e8ffd6ba15cf576d9cf5a1e4962d9c7629f69ee3fd83ed388ff195"}
Oct 30 22:18:33.651195 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.651127    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-xkcch" podStartSLOduration=311.375209799 podStartE2EDuration="5m15.651108265s" totalImagesPullingTime="4.275898466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:33.524238201 +0000 UTC m=+35.911506046" watchObservedRunningTime="2026-10-30 22:18:33.651108265 +0000 UTC m=+36.038376117"
Oct 30 22:18:33.652724 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.652691    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6mwl7"]
Oct 30 22:18:33.668963 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.668928    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6mwl7"]
Oct 30 22:18:33.669084 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.669032    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.673617 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.673440    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-qstj7-lntsk"]
Oct 30 22:18:33.680692 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.680624    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:18:33.680910 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.680874    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:18:33.683648 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.683614    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-srblb\""
Oct 30 22:18:33.687815 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.687759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.691050 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.691008    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qstj7-lntsk"]
Oct 30 22:18:33.780729 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.780697    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dhvzl\""
Oct 30 22:18:33.791125 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.791086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wnl2\" (UniqueName: \"kubernetes.io/projected/9d45821a-5cff-41e5-b50c-90940f18d7c9-kube-api-access-6wnl2\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.791285 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.791145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-trusted-ca-bundle\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.791285 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.791251    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9sn7\" (UniqueName: \"kubernetes.io/projected/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-api-access-t9sn7\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.791358 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.791297    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-service-ca-bundle\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.791392 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.791352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3d84917-1245-4b8c-a887-fd2d668bcd2d-data-volume\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.791423 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.791413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.791476 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.791459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.791549 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.791492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d45821a-5cff-41e5-b50c-90940f18d7c9-archives-path\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.791623 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.791551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3d84917-1245-4b8c-a887-fd2d668bcd2d-crio-socket\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.892483 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.892444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3d84917-1245-4b8c-a887-fd2d668bcd2d-data-volume\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.892663 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.892582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.892663 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.892640    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.892663 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.892660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d45821a-5cff-41e5-b50c-90940f18d7c9-archives-path\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.892832 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.892687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3d84917-1245-4b8c-a887-fd2d668bcd2d-crio-socket\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.892832 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.892728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wnl2\" (UniqueName: \"kubernetes.io/projected/9d45821a-5cff-41e5-b50c-90940f18d7c9-kube-api-access-6wnl2\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.892832 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.892747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-trusted-ca-bundle\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.892832 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.892805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9sn7\" (UniqueName: \"kubernetes.io/projected/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-api-access-t9sn7\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.892832 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.892831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-service-ca-bundle\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.893534 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.893465    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-service-ca-bundle\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.893785 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.893764    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3d84917-1245-4b8c-a887-fd2d668bcd2d-data-volume\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.893855 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:33.893832    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:18:33.893917 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:33.893888    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:34.393870198 +0000 UTC m=+36.781138028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : configmap "kube-rbac-proxy" not found
Oct 30 22:18:33.893974 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:33.893967    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:33.894029 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:33.894000    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:34.393988031 +0000 UTC m=+36.781255857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:33.894324 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.894207    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d45821a-5cff-41e5-b50c-90940f18d7c9-archives-path\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.894324 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.894264    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3d84917-1245-4b8c-a887-fd2d668bcd2d-crio-socket\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.895494 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.895392    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-trusted-ca-bundle\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:33.910926 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.910901    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9sn7\" (UniqueName: \"kubernetes.io/projected/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-api-access-t9sn7\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:33.916226 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:33.916180    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wnl2\" (UniqueName: \"kubernetes.io/projected/9d45821a-5cff-41e5-b50c-90940f18d7c9-kube-api-access-6wnl2\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:34.079466 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.079432    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:34.398814 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.398741    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:34.398814 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.398798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:34.399047 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:34.399030    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:34.399096 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:34.399080    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:35.399066312 +0000 UTC m=+37.786334149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:34.399460 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:34.399443    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:18:34.399561 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:34.399493    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:35.399480217 +0000 UTC m=+37.786748053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : configmap "kube-rbac-proxy" not found
Oct 30 22:18:34.404529 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.404485    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qstj7-lntsk"]
Oct 30 22:18:34.422317 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:34.422285    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d45821a_5cff_41e5_b50c_90940f18d7c9.slice/crio-52e1662a1fb196c8242ec74e44da518dadad25943143c13f8984fdcd7795a6d4 WatchSource:0}: Error finding container 52e1662a1fb196c8242ec74e44da518dadad25943143c13f8984fdcd7795a6d4: Status 404 returned error can't find the container with id 52e1662a1fb196c8242ec74e44da518dadad25943143c13f8984fdcd7795a6d4
Oct 30 22:18:34.494850 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.494371    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h" event={"ID":"9f16a8be-4c89-4786-a385-ff738661c9bb","Type":"ContainerStarted","Data":"ff941d3d7ef8aefa21841303705314ad672e95364bd7943e557c850d759dba53"}
Oct 30 22:18:34.498358 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.498323    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qstj7-lntsk" event={"ID":"9d45821a-5cff-41e5-b50c-90940f18d7c9","Type":"ContainerStarted","Data":"52e1662a1fb196c8242ec74e44da518dadad25943143c13f8984fdcd7795a6d4"}
Oct 30 22:18:34.507653 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.506786    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/1.log"
Oct 30 22:18:34.507946 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.507927    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/0.log"
Oct 30 22:18:34.507999 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.507966    2583 generic.go:356] "Generic (PLEG): container finished" podID="88c3c753-4382-452d-898d-732766a65bf1" containerID="6b06fc816b5d2acc17993f19f7a4cabafe3351bda1a804c24f06f570d748d8af" exitCode=255
Oct 30 22:18:34.508357 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.508341    2583 scope.go:122] "RemoveContainer" containerID="6b06fc816b5d2acc17993f19f7a4cabafe3351bda1a804c24f06f570d748d8af"
Oct 30 22:18:34.508596 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:34.508569    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dd7n4_openshift-console-operator(88c3c753-4382-452d-898d-732766a65bf1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" podUID="88c3c753-4382-452d-898d-732766a65bf1"
Oct 30 22:18:34.508735 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.508712    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" event={"ID":"88c3c753-4382-452d-898d-732766a65bf1","Type":"ContainerDied","Data":"6b06fc816b5d2acc17993f19f7a4cabafe3351bda1a804c24f06f570d748d8af"}
Oct 30 22:18:34.508811 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.508770    2583 scope.go:122] "RemoveContainer" containerID="370f815774aa68d2abc729082052137182679357706cf311925c92831d1b9a11"
Oct 30 22:18:34.511906 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.511866    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b" event={"ID":"f8199465-5b50-41fd-ab4f-5201977736a3","Type":"ContainerStarted","Data":"f2d55e4dd5911f41dd623deb5c81a34971d1eadc1c3c140e8e0c840be55f0de3"}
Oct 30 22:18:34.523042 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.522993    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h" podStartSLOduration=311.761874159 podStartE2EDuration="5m16.52297541s" totalImagesPullingTime="4.761101251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:34.52165814 +0000 UTC m=+36.908925985" watchObservedRunningTime="2026-10-30 22:18:34.52297541 +0000 UTC m=+36.910243254"
Oct 30 22:18:34.542084 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.540813    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b" podStartSLOduration=311.74630962 podStartE2EDuration="5m16.540794985s" totalImagesPullingTime="4.794485365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:34.539789814 +0000 UTC m=+36.927057660" watchObservedRunningTime="2026-10-30 22:18:34.540794985 +0000 UTC m=+36.928062832"
Oct 30 22:18:34.907403 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.907372    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qlkzv"]
Oct 30 22:18:34.931703 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.931667    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qlkzv"]
Oct 30 22:18:34.931834 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.931786    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:34.936744 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.936424    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vjg9f\""
Oct 30 22:18:34.936744 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.936548    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 22:18:34.936744 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.936650    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 22:18:34.936744 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.936692    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 22:18:34.936980 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:34.936803    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 22:18:35.010929 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.010895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-cabundle\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:35.011111 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.011086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-295mn\" (UniqueName: \"kubernetes.io/projected/afc45de8-4044-44c6-ac24-c39d8b2eece5-kube-api-access-295mn\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:35.011179 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.011156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-key\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:35.126113 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.126016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295mn\" (UniqueName: \"kubernetes.io/projected/afc45de8-4044-44c6-ac24-c39d8b2eece5-kube-api-access-295mn\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:35.126113 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.126100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-key\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:35.126318 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.126218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-cabundle\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:35.127025 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.127002    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-cabundle\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:35.131310 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.131283    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-key\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:35.139837 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.139786    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-295mn\" (UniqueName: \"kubernetes.io/projected/afc45de8-4044-44c6-ac24-c39d8b2eece5-kube-api-access-295mn\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:35.158549 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.158518    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j"]
Oct 30 22:18:35.172058 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.172030    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd"]
Oct 30 22:18:35.172314 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.172288    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j"
Oct 30 22:18:35.179360 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.179338    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9pq22\""
Oct 30 22:18:35.179484 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.179375    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 22:18:35.180664 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.180632    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:35.188335 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.188257    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j"]
Oct 30 22:18:35.188335 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.188297    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd"]
Oct 30 22:18:35.188335 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.188324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd"
Oct 30 22:18:35.262475 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.262438    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:35.327784 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.327760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdl48\" (UniqueName: \"kubernetes.io/projected/c097d835-0be1-4736-8dd8-98f375f02f47-kube-api-access-qdl48\") pod \"migrator-66df5d496b-vfc6j\" (UID: \"c097d835-0be1-4736-8dd8-98f375f02f47\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j"
Oct 30 22:18:35.327859 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.327812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpzp6\" (UniqueName: \"kubernetes.io/projected/16e7c643-41ee-4ac5-b285-1351ae323a1d-kube-api-access-mpzp6\") pod \"migrator-66df5d496b-ds7bd\" (UID: \"16e7c643-41ee-4ac5-b285-1351ae323a1d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd"
Oct 30 22:18:35.429099 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.429024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:35.429099 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.429067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdl48\" (UniqueName: \"kubernetes.io/projected/c097d835-0be1-4736-8dd8-98f375f02f47-kube-api-access-qdl48\") pod \"migrator-66df5d496b-vfc6j\" (UID: \"c097d835-0be1-4736-8dd8-98f375f02f47\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j"
Oct 30 22:18:35.429299 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.429109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:35.429299 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.429164    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:18:35.429299 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.429177    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpzp6\" (UniqueName: \"kubernetes.io/projected/16e7c643-41ee-4ac5-b285-1351ae323a1d-kube-api-access-mpzp6\") pod \"migrator-66df5d496b-ds7bd\" (UID: \"16e7c643-41ee-4ac5-b285-1351ae323a1d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd"
Oct 30 22:18:35.429299 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.429210    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:35.429299 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.429261    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:37.429246599 +0000 UTC m=+39.816514443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:35.429299 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.429278    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:37.429268707 +0000 UTC m=+39.816536547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : configmap "kube-rbac-proxy" not found
Oct 30 22:18:35.443524 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.443224    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpzp6\" (UniqueName: \"kubernetes.io/projected/16e7c643-41ee-4ac5-b285-1351ae323a1d-kube-api-access-mpzp6\") pod \"migrator-66df5d496b-ds7bd\" (UID: \"16e7c643-41ee-4ac5-b285-1351ae323a1d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd"
Oct 30 22:18:35.443524 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.443462    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdl48\" (UniqueName: \"kubernetes.io/projected/c097d835-0be1-4736-8dd8-98f375f02f47-kube-api-access-qdl48\") pod \"migrator-66df5d496b-vfc6j\" (UID: \"c097d835-0be1-4736-8dd8-98f375f02f47\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j"
Oct 30 22:18:35.504016 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.503993    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j"
Oct 30 22:18:35.517142 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.517112    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rvz8k" event={"ID":"188a085b-a512-440b-90ec-30acc66e4171","Type":"ContainerStarted","Data":"65443a4435f1f4f443e07d536538c12382f9d6b10b0ab1017da4e4a091b99bb0"}
Oct 30 22:18:35.518802 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.518778    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6" event={"ID":"53de1b04-dea0-4a6f-9609-760792cc0406","Type":"ContainerStarted","Data":"b7d6941b3bca844bc3b63df1c3a339f8f1f282a9bbc269e92c7f5bb1bf427c2a"}
Oct 30 22:18:35.520216 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.520187    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qstj7-lntsk" event={"ID":"9d45821a-5cff-41e5-b50c-90940f18d7c9","Type":"ContainerStarted","Data":"c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442"}
Oct 30 22:18:35.521856 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.521839    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/1.log"
Oct 30 22:18:35.522128 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.522114    2583 scope.go:122] "RemoveContainer" containerID="6b06fc816b5d2acc17993f19f7a4cabafe3351bda1a804c24f06f570d748d8af"
Oct 30 22:18:35.522287 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.522270    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dd7n4_openshift-console-operator(88c3c753-4382-452d-898d-732766a65bf1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" podUID="88c3c753-4382-452d-898d-732766a65bf1"
Oct 30 22:18:35.534406 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.534386    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd"
Oct 30 22:18:35.541177 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.540411    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rvz8k" podStartSLOduration=2.739872853 podStartE2EDuration="8.540394807s" totalImagesPullingTime="5.800521954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:35.5390897 +0000 UTC m=+37.926357549" watchObservedRunningTime="2026-10-30 22:18:35.540394807 +0000 UTC m=+37.927662654"
Oct 30 22:18:35.559998 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.559944    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-qstj7-lntsk" podStartSLOduration=2.559931311 podStartE2EDuration="2.559931311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:35.556541634 +0000 UTC m=+37.943809479" watchObservedRunningTime="2026-10-30 22:18:35.559931311 +0000 UTC m=+37.947199156"
Oct 30 22:18:35.577258 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.577055    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qlkzv"]
Oct 30 22:18:35.579755 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.579716    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6" podStartSLOduration=306.348840615 podStartE2EDuration="5m12.579704859s" totalImagesPullingTime="6.230864244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:35.579163572 +0000 UTC m=+37.966431419" watchObservedRunningTime="2026-10-30 22:18:35.579704859 +0000 UTC m=+37.966972705"
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.731698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.731744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.731773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.731800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.731833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.731884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.731921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.732005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732142    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732181    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert podName:88d59947-036f-412d-9ee6-f7565b0d4d8b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:43.732168139 +0000 UTC m=+46.119435963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-grpzl" (UID: "88d59947-036f-412d-9ee6-f7565b0d4d8b") : secret "networking-console-plugin-cert" not found
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732220    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732241    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs podName:99252037-fc1d-4055-b66b-2f0e8ab910b8 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:43.73223472 +0000 UTC m=+46.119502544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs") pod "router-default-6668ff59b7-vglrn" (UID: "99252037-fc1d-4055-b66b-2f0e8ab910b8") : secret "router-metrics-certs-default" not found
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732290    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:35.732628 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732316    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls podName:c69ae355-a045-48d1-b7c9-881d2c4e8a1b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:43.732309363 +0000 UTC m=+46.119577190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4vs7k" (UID: "c69ae355-a045-48d1-b7c9-881d2c4e8a1b") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732356    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle podName:99252037-fc1d-4055-b66b-2f0e8ab910b8 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:43.732350984 +0000 UTC m=+46.119618808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle") pod "router-default-6668ff59b7-vglrn" (UID: "99252037-fc1d-4055-b66b-2f0e8ab910b8") : configmap references non-existent config key: service-ca.crt
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732394    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732401    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bfb9545c9-tkcrm: secret "image-registry-tls" not found
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732419    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls podName:2d4504af-560b-4e5a-a3cf-befff8d8d6d2 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:43.732414192 +0000 UTC m=+46.119682017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls") pod "image-registry-5bfb9545c9-tkcrm" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2") : secret "image-registry-tls" not found
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732448    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732475    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls podName:83a06c56-145a-4a52-baca-2d01c679e581 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:43.732466879 +0000 UTC m=+46.119734709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r6fws" (UID: "83a06c56-145a-4a52-baca-2d01c679e581") : secret "samples-operator-tls" not found
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732540    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732547    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5559797b78-pnvl9: secret "image-registry-tls" not found
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732565    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls podName:555a97b6-857c-4374-8939-1eed2d487b4c nodeName:}" failed. No retries permitted until 2026-10-30 22:18:43.732559531 +0000 UTC m=+46.119827355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls") pod "image-registry-5559797b78-pnvl9" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c") : secret "image-registry-tls" not found
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732598    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:18:35.733145 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:35.732614    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls podName:8e71c29d-cd72-42ae-a0de-1500b78c89b0 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:43.732609313 +0000 UTC m=+46.119877137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls") pod "dns-default-6wrrt" (UID: "8e71c29d-cd72-42ae-a0de-1500b78c89b0") : secret "dns-default-metrics-tls" not found
Oct 30 22:18:35.824522 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.824475    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j"]
Oct 30 22:18:35.827121 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:35.827097    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc097d835_0be1_4736_8dd8_98f375f02f47.slice/crio-9aed098c2b53ad0567a37b484ceec5bce0bad98ee0cc8f230ce74329d7f4b3ab WatchSource:0}: Error finding container 9aed098c2b53ad0567a37b484ceec5bce0bad98ee0cc8f230ce74329d7f4b3ab: Status 404 returned error can't find the container with id 9aed098c2b53ad0567a37b484ceec5bce0bad98ee0cc8f230ce74329d7f4b3ab
Oct 30 22:18:35.855697 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.855499    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd"]
Oct 30 22:18:35.858881 ip-10-0-138-251 kubenswrapper[2583]: W1030 22:18:35.858842    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e7c643_41ee_4ac5_b285_1351ae323a1d.slice/crio-5cb04f796d45f2f400dfbc9c0ac25fb7aad2c369b0e8df3d6264d6d2240d0e7c WatchSource:0}: Error finding container 5cb04f796d45f2f400dfbc9c0ac25fb7aad2c369b0e8df3d6264d6d2240d0e7c: Status 404 returned error can't find the container with id 5cb04f796d45f2f400dfbc9c0ac25fb7aad2c369b0e8df3d6264d6d2240d0e7c
Oct 30 22:18:35.946175 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:35.946141    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qqwss_92a4974a-3e6e-4575-b5bd-303beb3fa201/dns-node-resolver/0.log"
Oct 30 22:18:36.532834 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:36.532767    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd" event={"ID":"16e7c643-41ee-4ac5-b285-1351ae323a1d","Type":"ContainerStarted","Data":"5cb04f796d45f2f400dfbc9c0ac25fb7aad2c369b0e8df3d6264d6d2240d0e7c"}
Oct 30 22:18:36.535119 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:36.535027    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j" event={"ID":"c097d835-0be1-4736-8dd8-98f375f02f47","Type":"ContainerStarted","Data":"9aed098c2b53ad0567a37b484ceec5bce0bad98ee0cc8f230ce74329d7f4b3ab"}
Oct 30 22:18:36.539317 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:36.538906    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv" event={"ID":"afc45de8-4044-44c6-ac24-c39d8b2eece5","Type":"ContainerStarted","Data":"129a50dcd517ece883ffae717801ea649f0cc077c7c58b0ef5d8640259372253"}
Oct 30 22:18:36.539317 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:36.538938    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv" event={"ID":"afc45de8-4044-44c6-ac24-c39d8b2eece5","Type":"ContainerStarted","Data":"d739400426ebca3446563046bb9acb55606de90d38407363c8032e47bb1a87d2"}
Oct 30 22:18:36.546209 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:36.546011    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tgn5k" event={"ID":"8eab1d96-88bb-417a-b03e-8e2f0274e862","Type":"ContainerStarted","Data":"4c08006e29bc14963978214fb234420ca1eb0e69947746d460768c0a257b01b1"}
Oct 30 22:18:36.546209 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:36.546175    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:36.575530 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:36.574808    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv" podStartSLOduration=2.574777669 podStartE2EDuration="2.574777669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:36.57343729 +0000 UTC m=+38.960705148" watchObservedRunningTime="2026-10-30 22:18:36.574777669 +0000 UTC m=+38.962045710"
Oct 30 22:18:36.741978 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:36.741862    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7cjdl_a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6/node-ca/0.log"
Oct 30 22:18:37.452919 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:37.452883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:37.453108 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:37.452931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:37.453108 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:37.453030    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:18:37.453212 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:37.453114    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:41.453093657 +0000 UTC m=+43.840361487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : configmap "kube-rbac-proxy" not found
Oct 30 22:18:37.453212 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:37.453128    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:37.453212 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:37.453188    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:41.453171456 +0000 UTC m=+43.840439300 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:37.542445 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:37.542411    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rvz8k_188a085b-a512-440b-90ec-30acc66e4171/serve-healthcheck-canary/0.log"
Oct 30 22:18:38.285065 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:38.285028    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:38.285231 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:38.285090    2583 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:38.285524 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:38.285488    2583 scope.go:122] "RemoveContainer" containerID="6b06fc816b5d2acc17993f19f7a4cabafe3351bda1a804c24f06f570d748d8af"
Oct 30 22:18:38.285747 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:38.285725    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dd7n4_openshift-console-operator(88c3c753-4382-452d-898d-732766a65bf1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" podUID="88c3c753-4382-452d-898d-732766a65bf1"
Oct 30 22:18:38.556607 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:38.556521    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd" event={"ID":"16e7c643-41ee-4ac5-b285-1351ae323a1d","Type":"ContainerStarted","Data":"8cc66f70939d80064efebad4881422ef32b2b32ea915ebfe5d519dba600b79c6"}
Oct 30 22:18:38.557899 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:38.557867    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j" event={"ID":"c097d835-0be1-4736-8dd8-98f375f02f47","Type":"ContainerStarted","Data":"e744a580e03d4e5d053bc643fe7780f557632b731b6d4ee2358a7a7bd6ea338b"}
Oct 30 22:18:38.558302 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:38.558286    2583 scope.go:122] "RemoveContainer" containerID="6b06fc816b5d2acc17993f19f7a4cabafe3351bda1a804c24f06f570d748d8af"
Oct 30 22:18:38.558431 ip-10-0-138-251 kubenswrapper[2583]: E1030 22:18:38.558414    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dd7n4_openshift-console-operator(88c3c753-4382-452d-898d-732766a65bf1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" podUID="88c3c753-4382-452d-898d-732766a65bf1"
Oct 30 22:18:38.589610 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:38.589573    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tgn5k" podStartSLOduration=37.526674397 podStartE2EDuration="40.589562723s" totalImagesPullingTime="3.062888326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:36.594614265 +0000 UTC m=+38.981882111" watchObservedRunningTime="2026-10-30 22:18:38.589562723 +0000 UTC m=+40.976830568"
Oct 30 22:18:38.589886 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:38.589869    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd" podStartSLOduration=1.6741213830000001 podStartE2EDuration="3.589865266s" totalImagesPullingTime="1.915743883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:38.588581391 +0000 UTC m=+40.975849234" watchObservedRunningTime="2026-10-30 22:18:38.589865266 +0000 UTC m=+40.977133112"
Oct 30 22:18:38.612205 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:38.612165    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j" podStartSLOduration=1.6674953609999998 podStartE2EDuration="3.6121545s" totalImagesPullingTime="1.944659139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:38.611048697 +0000 UTC m=+40.998316543" watchObservedRunningTime="2026-10-30 22:18:38.6121545 +0000 UTC m=+40.999422347"
Oct 30 22:18:39.083788 ip-10-0-138-251 kubenswrapper[2583]: I1030 22:18:39.083650    2583 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:18:39.083812 ip-10-0-138-251 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 22:18:39.105929 ip-10-0-138-251 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 22:18:39.106096 ip-10-0-138-251 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 22:18:39.106145 ip-10-0-138-251 systemd[1]: kubelet.service: Consumed 3.160s CPU time.
Oct 30 22:18:39.116069 ip-10-0-138-251 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:18:39.209457 ip-10-0-138-251 kubenswrapper[5060]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:18:39.209457 ip-10-0-138-251 kubenswrapper[5060]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:18:39.209457 ip-10-0-138-251 kubenswrapper[5060]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:18:39.209457 ip-10-0-138-251 kubenswrapper[5060]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:18:39.211483 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211402    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:18:39.211483 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211482    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:18:39.211483 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211486    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211490    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211492    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211495    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211498    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211500    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211516    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211520    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211522    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211525    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211528    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211531    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211534    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211536    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211539    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211542    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211545    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211547    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211550    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211552    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:18:39.211592 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211555    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211557    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211560    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211563    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211566    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211576    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211579    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211582    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211585    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211587    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211590    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211593    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211596    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211599    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211601    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211604    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211607    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211610    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211613    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:18:39.212061 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211615    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211620    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211624    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211628    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211630    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211633    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211636    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211639    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211642    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211645    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211648    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211650    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211653    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211655    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211658    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211661    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211665    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211668    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211671    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:18:39.212521 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211674    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211676    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211679    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211682    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211684    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211686    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211689    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211692    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211695    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211697    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211700    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211702    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211705    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211708    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211711    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211716    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211719    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211721    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211763    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211766    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:18:39.212966 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211769    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211772    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211775    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211777    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211780    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211782    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211785    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211788    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211790    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211793    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211795    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211798    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211800    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211803    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211806    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211809    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211811    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211815    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211817    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211820    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:18:39.213435 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211823    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211825    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211828    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211831    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211834    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.211836    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212591    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212601    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212603    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212607    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212609    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212612    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212614    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212617    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212620    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212622    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212625    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212627    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212630    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212632    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:18:39.213935 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212635    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212638    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212640    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212643    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212645    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212648    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212650    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212653    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212656    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212658    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212661    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212664    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212670    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212673    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212676    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212678    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212681    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212683    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212690    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212693    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:18:39.214406 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212697    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212701    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212704    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212707    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212709    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212712    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212714    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212717    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212720    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212723    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212725    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212728    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212730    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212733    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212735    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212738    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212740    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212743    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212745    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:18:39.214913 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212748    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212751    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212753    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212756    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212759    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212764    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212767    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212769    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212772    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212774    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212777    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212780    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212783    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212785    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212788    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212791    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212794    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212796    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212801    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:18:39.215362 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212805    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212808    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212811    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212813    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212816    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212819    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212822    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212825    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212827    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212830    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212832    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212835    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212837    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212840    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212843    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212845    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212848    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212850    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212855    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:18:39.215816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212857    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212860    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212862    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212865    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212867    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212870    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212874    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212877    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212879    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212882    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212885    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212887    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212890    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212892    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.212896    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213350    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213355    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213357    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213360    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213363    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:18:39.216302 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213365    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213368    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213371    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213373    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213376    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213378    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213381    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213384    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213386    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213389    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213391    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213395    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213397    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213400    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213402    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213405    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213407    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213410    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213414    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213417    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:18:39.216791 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213420    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213422    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213425    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213427    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213430    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213432    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213435    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213438    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213440    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213443    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213445    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213448    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213451    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213453    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213456    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213458    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213461    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213463    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213466    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:18:39.217273 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213468    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213471    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213474    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213478    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213482    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213484    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213487    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213491    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213495    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213498    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213501    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213518    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213521    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213524    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213526    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213529    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213532    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213534    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213537    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:18:39.217725 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213540    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213542    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213545    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213549    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213552    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213555    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213558    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213561    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213564    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213566    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213568    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213571    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213574    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213576    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213578    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213581    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213584    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213586    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213589    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213591    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:18:39.218166 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213594    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213596    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213599    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213601    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213605    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213607    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213610    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213613    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213616    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213618    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213621    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213623    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213626    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213629    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213631    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213634    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213636    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213639    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213642    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213645    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:18:39.218651 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213648    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213650    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.213653    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213738    5060 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213745    5060 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213750    5060 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213755    5060 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213759    5060 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213763    5060 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213767    5060 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213771    5060 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213774    5060 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213779    5060 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213782    5060 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213785    5060 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213788    5060 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213793    5060 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213796    5060 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213799    5060 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213802    5060 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213806    5060 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213810    5060 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213813    5060 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:18:39.219257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213816    5060 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213819    5060 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213822    5060 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213826    5060 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213829    5060 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213833    5060 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213836    5060 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213839    5060 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213842    5060 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213845    5060 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213848    5060 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213851    5060 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213856    5060 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213859    5060 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213861    5060 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213864    5060 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213867    5060 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213871    5060 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213874    5060 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213877    5060 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213881    5060 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213888    5060 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213892    5060 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213899    5060 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:18:39.219811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213903    5060 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213909    5060 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213914    5060 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213919    5060 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213922    5060 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213925    5060 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213929    5060 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213932    5060 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213935    5060 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213938    5060 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213943    5060 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213946    5060 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213949    5060 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213952    5060 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213956    5060 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213959    5060 flags.go:64] FLAG: --help="false"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213962    5060 flags.go:64] FLAG: --hostname-override="ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213965    5060 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213968    5060 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213971    5060 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213975    5060 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213978    5060 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213982    5060 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:18:39.220364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213985    5060 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213988    5060 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213991    5060 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213993    5060 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.213997    5060 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214000    5060 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214004    5060 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214007    5060 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214010    5060 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214013    5060 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214018    5060 flags.go:64] FLAG: --lock-file=""
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214020    5060 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214024    5060 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214027    5060 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214032    5060 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214035    5060 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214039    5060 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214042    5060 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214045    5060 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214048    5060 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214051    5060 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214054    5060 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214059    5060 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214062    5060 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214066    5060 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:18:39.220944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214069    5060 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214072    5060 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214075    5060 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214078    5060 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214081    5060 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214084    5060 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214087    5060 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214094    5060 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214097    5060 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214100    5060 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214103    5060 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214106    5060 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214109    5060 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214113    5060 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214116    5060 flags.go:64] FLAG: --port="10250"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214119    5060 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214122    5060 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-078c84db754bfab61"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214127    5060 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214130    5060 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214133    5060 flags.go:64] FLAG: --register-node="true"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214136    5060 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214140    5060 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214143    5060 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214146    5060 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214161    5060 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214165    5060 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:18:39.221662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214168    5060 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214172    5060 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214174    5060 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214177    5060 flags.go:64] FLAG: --runonce="false"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214180    5060 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214183    5060 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214187    5060 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214189    5060 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214192    5060 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214195    5060 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214198    5060 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214202    5060 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214204    5060 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214207    5060 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214210    5060 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214213    5060 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214216    5060 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214220    5060 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214222    5060 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214230    5060 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214233    5060 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214236    5060 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214240    5060 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214245    5060 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214247    5060 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:18:39.222663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214250    5060 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214253    5060 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214257    5060 flags.go:64] FLAG: --v="2"
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214261    5060 flags.go:64] FLAG: --version="false"
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214265    5060 flags.go:64] FLAG: --vmodule=""
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214269    5060 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214273    5060 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214421    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214425    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214429    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214432    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214435    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214437    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214440    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214442    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214445    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214449    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214451    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214454    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214457    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214459    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:18:39.223335 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214462    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214464    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214467    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214469    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214472    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214479    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214482    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214485    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214488    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214491    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214494    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214497    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214499    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214502    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214517    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214522    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214525    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214529    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214534    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214538    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:18:39.223914 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214542    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214545    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214547    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214550    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214553    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214555    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214558    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214560    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214563    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214565    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214568    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214570    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214573    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214575    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214578    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214581    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214583    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214588    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214590    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:18:39.224706 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214593    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214595    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214598    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214600    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214603    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214605    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214608    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214610    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214613    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214615    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214618    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214620    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214630    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214632    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214635    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214638    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214641    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214643    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214646    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214648    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:18:39.225188 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214651    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214653    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214656    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214658    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214661    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214663    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214666    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214668    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214671    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214673    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214678    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214682    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214684    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214687    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214690    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214692    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214695    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214697    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214699    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:18:39.225699 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214702    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214706    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214710    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214714    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214716    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214719    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214722    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214725    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214727    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214730    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214732    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214735    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214737    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214739    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.214765    5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214798    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:18:39.226142 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214802    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214806    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214808    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214811    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214813    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214816    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214818    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214822    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214825    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214827    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214830    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214832    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214834    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214837    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214839    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214842    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214844    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214847    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214850    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214854    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:18:39.231669 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214857    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214859    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214862    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214865    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214867    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214870    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214872    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214875    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214878    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214880    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214883    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214885    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214889    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214893    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214895    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214898    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214901    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214903    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214906    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:18:39.232168 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214910    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214912    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214915    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214919    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214923    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214925    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214928    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214930    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214933    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214935    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214938    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214940    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214945    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214948    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214951    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214953    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214956    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214959    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214962    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:18:39.232668 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214964    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214967    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214969    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214972    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214974    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214977    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214980    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214982    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214985    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214987    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214990    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214992    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214995    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.214998    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215001    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215003    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215006    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215009    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215011    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215014    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:18:39.233107 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215016    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215018    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215021    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215023    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215026    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215030    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215033    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215035    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215038    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215040    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215043    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215046    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215048    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215051    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215053    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215056    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215058    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215061    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215064    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:18:39.233576 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215066    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215069    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215071    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215074    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215076    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215079    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215083    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.215086    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.218696    5060 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.218711    5060 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.218729    5060 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.218734    5060 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218787    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218793    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218796    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218800    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218802    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218805    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218808    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:18:39.234009 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218810    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218813    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218816    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218819    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218821    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218827    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218830    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218833    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218835    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218838    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218841    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218844    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218847    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218849    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218852    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218854    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218857    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218861    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218865    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:18:39.234457 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218867    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218871    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218874    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218877    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218881    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218885    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218887    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218890    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218893    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218895    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218898    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218901    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218904    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218906    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218909    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218912    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218914    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218916    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218919    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:18:39.234920 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218922    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218924    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218927    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218930    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218933    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218935    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218938    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218940    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218944    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218947    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218949    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218952    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218954    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218957    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218960    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218962    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218964    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218967    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218969    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218972    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:18:39.235371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218975    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218978    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218980    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218983    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218987    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218991    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218994    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.218997    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219000    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219002    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219005    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219008    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219010    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219013    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219015    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219018    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219021    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219023    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219026    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:18:39.235855 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219029    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219032    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219035    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219037    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219040    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219042    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219045    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219048    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219050    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219053    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219055    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219058    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219060    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219063    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219065    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219068    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219071    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219073    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219076    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219078    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:18:39.236294 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219081    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219083    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.219108    5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219268    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219273    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219276    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219279    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219283    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219286    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219288    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219291    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219294    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219297    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219300    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219303    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219305    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:18:39.236763 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219308    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219310    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219313    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219316    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219318    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219320    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219323    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219326    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219328    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219331    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219333    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219336    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219339    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219341    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219344    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219346    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219349    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219351    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219354    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219356    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:18:39.237150 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219359    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219361    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219364    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219366    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219370    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219372    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219375    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219378    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219380    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219383    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219385    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219388    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219390    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219392    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219395    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219397    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219400    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219402    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219405    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:18:39.237709 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219407    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219421    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219424    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219428    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219430    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219433    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219435    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219437    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219440    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219443    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219445    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219447    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219450    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219452    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219455    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219457    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219460    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219463    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219466    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:18:39.238187 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219469    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219471    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219474    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219476    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219479    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219481    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219484    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219486    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219489    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219491    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219494    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219498    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219501    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219515    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219520    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219523    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219527    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219530    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219533    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:18:39.238936 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219535    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219538    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219541    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219543    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219546    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219548    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219551    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219553    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219556    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219558    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219561    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219563    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219566    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219569    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219571    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219574    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:18:39.239816 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.219597    5060 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219630    5060 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219635    5060 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219638    5060 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219641    5060 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219643    5060 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219646    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219648    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219651    5060 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219655    5060 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219658    5060 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219660    5060 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219663    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219666    5060 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219668    5060 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219671    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219674    5060 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219677    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219679    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:18:39.240425 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219682    5060 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219685    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219687    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219689    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219692    5060 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219694    5060 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219697    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219700    5060 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219702    5060 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219705    5060 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219708    5060 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219711    5060 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219713    5060 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219716    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219718    5060 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219721    5060 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219724    5060 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219726    5060 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219728    5060 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219731    5060 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:18:39.240969 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219735    5060 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219739    5060 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219744    5060 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219748    5060 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219751    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219754    5060 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219757    5060 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219760    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219763    5060 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219766    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219769    5060 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219771    5060 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219774    5060 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219776    5060 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219779    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219781    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219784    5060 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219787    5060 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219789    5060 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:18:39.241441 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219792    5060 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219794    5060 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219797    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219799    5060 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219801    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219805    5060 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219807    5060 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219810    5060 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219813    5060 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219816    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219819    5060 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219821    5060 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219824    5060 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219826    5060 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219829    5060 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219832    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219835    5060 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219837    5060 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219840    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219842    5060 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:18:39.241943 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219845    5060 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219847    5060 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219850    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219852    5060 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219855    5060 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219857    5060 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219860    5060 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219863    5060 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219865    5060 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219868    5060 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219870    5060 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219873    5060 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219875    5060 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219878    5060 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219880    5060 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219883    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219886    5060 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219888    5060 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219891    5060 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219894    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:18:39.242427 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219896    5060 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219899    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219902    5060 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219904    5060 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219907    5060 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219909    5060 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219912    5060 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219914    5060 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:39.219917    5060 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.219963    5060 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.220872    5060 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.220923    5060 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.221089    5060 server.go:1014] "Starting client certificate rotation"
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.221246    5060 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.221313    5060 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 22:12:58 +0000 UTC" deadline="2028-08-11 18:13:26.790950202 +0000 UTC"
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.221353    5060 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15619h54m47.569600516s"
Oct 30 22:18:39.242931 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.221538    5060 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:18:39.243316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.223724    5060 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:18:39.243316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.224996    5060 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:18:39.243316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.226616    5060 log.go:25] "Validated CRI v1 image API"
Oct 30 22:18:39.243316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.227327    5060 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:18:39.243316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.229226    5060 fs.go:135] Filesystem UUIDs: map[6c422d92-e786-44e4-922d-e390a0764faf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8948b817-8f27-4388-8be1-7a7cdc16a4c5:/dev/nvme0n1p3]
Oct 30 22:18:39.243477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.229249    5060 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00a6f191b21baf3cff1ae517e24ae8f98dad92eda7e8c5aa5e3d1a12f5c1179e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00a6f191b21baf3cff1ae517e24ae8f98dad92eda7e8c5aa5e3d1a12f5c1179e/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0509d0cf50d58ba7f871bd87079bee6a384612bcae5b2ed7c44494859ee66310/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0509d0cf50d58ba7f871bd87079bee6a384612bcae5b2ed7c44494859ee66310/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/064263fb24d1f1530e87ccd92388446309aac371f65d39927ceffeadc0e43185/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/064263fb24d1f1530e87ccd92388446309aac371f65d39927ceffeadc0e43185/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e69d09c541b7f90c5b73e2effd2c7846cf2d1059e71f811d091108f27cbfe01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e69d09c541b7f90c5b73e2effd2c7846cf2d1059e71f811d091108f27cbfe01/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24f6e5a3ac4313c52e40175843de822b11bd947e711492b5fb031d95fab873fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24f6e5a3ac4313c52e40175843de822b11bd947e711492b5fb031d95fab873fb/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ea75aed4060b21a6e5b4e68c6bbd8c4c0f466d2268d1bf40d2c983630c341be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ea75aed4060b21a6e5b4e68c6bbd8c4c0f466d2268d1bf40d2c983630c341be/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ebc14ac21416b778201bd1361cb164993cb192ee203cbc22c7419765d5202ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ebc14ac21416b778201bd1361cb164993cb192ee203cbc22c7419765d5202ce/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/458275610f6e084d5e480f1ef033cbdb6418b1d08834cd383adff7c4982a0715/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/458275610f6e084d5e480f1ef033cbdb6418b1d08834cd383adff7c4982a0715/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46263b706eaf4cfcc2d5ae28dacaa4a4ccd35a80ec5b8274f7f82a9e42799b0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46263b706eaf4cfcc2d5ae28dacaa4a4ccd35a80ec5b8274f7f82a9e42799b0b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46523a1fca3e6d49c6fd38149fad1450dded49837a8cd1edadaf4ab6ebbd07b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46523a1fca3e6d49c6fd38149fad1450dded49837a8cd1edadaf4ab6ebbd07b8/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49a89c7c4427f3075d63943fc81dd663470b547b2471beb6f7981c7bab57c811/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49a89c7c4427f3075d63943fc81dd663470b547b2471beb6f7981c7bab57c811/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52e1662a1fb196c8242ec74e44da518dadad25943143c13f8984fdcd7795a6d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52e1662a1fb196c8242ec74e44da518dadad25943143c13f8984fdcd7795a6d4/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53d20e4b5c3853d4a42880651f1ccd76d8168056eefce8b175a655fcbb8d6323/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53d20e4b5c3853d4a42880651f1ccd76d8168056eefce8b175a655fcbb8d6323/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cb04f796d45f2f400dfbc9c0ac25fb7aad2c369b0e8df3d6264d6d2240d0e7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cb04f796d45f2f400dfbc9c0ac25fb7aad2c369b0e8df3d6264d6d2240d0e7c/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6afb2c7f1be5f304d205eeb403dad888f0f128ab0260e09b4e54435c893ee416/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6afb2c7f1be5f304d205eeb403dad888f0f128ab0260e09b4e54435c893ee416/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83f8805eb5ebe70e2d9ebecf1c77dd7eb1aae9c735e184ff97480b2e56d9ad22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83f8805eb5ebe70e2d9ebecf1c77dd7eb1aae9c735e184ff97480b2e56d9ad22/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92b9a4460b7b1f4186ae630e84c9a29174ad749c12ff3b9640d144dfde700fc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92b9a4460b7b1f4186ae630e84c9a29174ad749c12ff3b9640d144dfde700fc6/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94be0e38e90de037403a5baf5200eab1e6a6d12380ad5ebd875bf62f85d7d53c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94be0e38e90de037403a5baf5200eab1e6a6d12380ad5ebd875bf62f85d7d53c/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9aed098c2b53ad0567a37b484ceec5bce0bad98ee0cc8f230ce74329d7f4b3ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9aed098c2b53ad0567a37b484ceec5bce0bad98ee0cc8f230ce74329d7f4b3ab/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac29696864f0f5cbe7d0bd530b05d9acbe4351b370c863f7a47b0cd3e2eefc15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac29696864f0f5cbe7d0bd530b05d9acbe4351b370c863f7a47b0cd3e2eefc15/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afa2b2cc445d9210c603695d7342906088b6a9ab69fe0776876ca02ce9505ffc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afa2b2cc445d9210c603695d7342906088b6a9ab69fe0776876ca02ce9505ffc/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d739400426ebca3446563046bb9acb55606de90d38407363c8032e47bb1a87d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d739400426ebca3446563046bb9acb55606de90d38407363c8032e47bb1a87d2/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e39b79b239aba08b4d306d91ee3e02b66c2352419b90dba87222d6a154cc462d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e39b79b239aba08b4d306d91ee3e02b66c2352419b90dba87222d6a154cc462d/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e55f947e1675bf5ae5340da6e81b272c1df6274582811d79d0f8defc375aad86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e55f947e1675bf5ae5340da6e81b272c1df6274582811d79d0f8defc375aad86/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec63bb01333f4ac171a9de591aa0030b6a7f80fd57913288b79fe49c64bbc597/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec63bb01333f4ac171a9de591aa0030b6a7f80fd57913288b79fe49c64bbc597/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc3cef33e0cca3abe20249fbbd2709a37c6793194d800f29a59f1e5955859074/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc3cef33e0cca3abe20249fbbd2709a37c6793194d800f29a59f1e5955859074/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/012d43c7-e3f2-4931-877e-cdf66eaf61b6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/012d43c7-e3f2-4931-877e-cdf66eaf61b6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/012d43c7-e3f2-4931-877e-cdf66eaf61b6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/012d43c7-e3f2-4931-877e-cdf66eaf61b6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/012d43c7-e3f2-4931-877e-cdf66eaf61b6/volumes/kubernetes.io~projected/kube-api-access-sgv2b:{mountpoint:/var/lib/kubelet/pods/012d43c7-e3f2-4931-877e-cdf66eaf61b6/volumes/kubernetes.io~projected/kube-api-access-sgv2b major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16e7c643-41ee-4ac5-b285-1351ae323a1d/volumes/kubernetes.io~projected/kube-api-access-mpzp6:{mountpoint:/var/lib/kubelet/pods/16e7c643-41ee-4ac5-b285-1351ae323a1d/volumes/kubernetes.io~projected/kube-api-access-mpzp6 major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16ece3f9-76fd-4c93-b915-d1c8cec664fb/volumes/kubernetes.io~projected/kube-api-access-nq7tw:{mountpoint:/var/lib/kubelet/pods/16ece3f9-76fd-4c93-b915-d1c8cec664fb/volumes/kubernetes.io~projected/kube-api-access-nq7tw major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16ece3f9-76fd-4c93-b915-d1c8cec664fb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/16ece3f9-76fd-4c93-b915-d1c8cec664fb/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1889bdc0-508e-4f97-80cb-4a5aa6e4f67f/volumes/kubernetes.io~projected/kube-api-access-9tndn:{mountpoint:/var/lib/kubelet/pods/1889bdc0-508e-4f97-80cb-4a5aa6e4f67f/volumes/kubernetes.io~projected/kube-api-access-9tndn major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/188a085b-a512-440b-90ec-30acc66e4171/volumes/kubernetes.io~projected/kube-api-access-86vmt:{mountpoint:/var/lib/kubelet/pods/188a085b-a512-440b-90ec-30acc66e4171/volumes/kubernetes.io~projected/kube-api-access-86vmt major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/188a085b-a512-440b-90ec-30acc66e4171/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/188a085b-a512-440b-90ec-30acc66e4171/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~projected/kube-api-access-rjxjz:{mountpoint:/var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~projected/kube-api-access-rjxjz major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e346445-18a9-494c-9b3f-6c0860f631f7/volumes/kubernetes.io~projected/kube-api-access-t8m5m:{mountpoint:/var/lib/kubelet/pods/2e346445-18a9-494c-9b3f-6c0860f631f7/volumes/kubernetes.io~projected/kube-api-access-t8m5m major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a7d925c-af25-4c40-a280-e870b59cb772/volumes/kubernetes.io~projected/kube-api-access-gq55l:{mountpoint:/var/lib/kubelet/pods/3a7d925c-af25-4c40-a280-e870b59cb772/volumes/kubernetes.io~projected/kube-api-access-gq55l major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/407cf470-1b82-424c-a8e2-a5f061b3b84c/volumes/kubernetes.io~projected/kube-api-access-7sfkt:{mountpoint:/var/lib/kubelet/pods/407cf470-1b82-424c-a8e2-a5f061b3b84c/volumes/kubernetes.io~projected/kube-api-access-7sfkt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/428997c4-b34e-4e3c-8551-c2d8510c69f4/volumes/kubernetes.io~projected/kube-api-access-kqzxg:{mountpoint:/var/lib/kubelet/pods/428997c4-b34e-4e3c-8551-c2d8510c69f4/volumes/kubernetes.io~projected/kube-api-access-kqzxg major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53de1b04-dea0-4a6f-9609-760792cc0406/volumes/kubernetes.io~projected/kube-api-access-f2x9v:{mountpoint:/var/lib/kubelet/pods/53de1b04-dea0-4a6f-9609-760792cc0406/volumes/kubernetes.io~projected/kube-api-access-f2x9v major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~projected/kube-api-access-xl7kc:{mountpoint:/var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~projected/kube-api-access-xl7kc major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60457f9c-65b0-4547-9012-2fc4efe2b9e8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/60457f9c-65b0-4547-9012-2fc4efe2b9e8/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a06c56-145a-4a52-baca-2d01c679e581/volumes/kubernetes.io~projected/kube-api-access-v87rd:{mountpoint:/var/lib/kubelet/pods/83a06c56-145a-4a52-baca-2d01c679e581/volumes/kubernetes.io~projected/kube-api-access-v87rd major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87c8cdc5-be2e-47a0-8cde-e944fda51056/volumes/kubernetes.io~projected/kube-api-access-wm75h:{mountpoint:/var/lib/kubelet/pods/87c8cdc5-be2e-47a0-8cde-e944fda51056/volumes/kubernetes.io~projected/kube-api-access-wm75h major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88c3c753-4382-452d-898d-732766a65bf1/volumes/kubernetes.io~projected/kube-api-access-j92gt:{mountpoint:/var/lib/kubelet/pods/88c3c753-4382-452d-898d-732766a65bf1/volumes/kubernetes.io~projected/kube-api-access-j92gt major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88c3c753-4382-452d-898d-732766a65bf1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/88c3c753-4382-452d-898d-732766a65bf1/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e71c29d-cd72-42ae-a0de-1500b78c89b0/volumes/kubernetes.io~projected/kube-api-access-9xd26:{mountpoint:/var/lib/kubelet/pods/8e71c29d-cd72-42ae-a0de-1500b78c89b0/volumes/kubernetes.io~projected/kube-api-access-9xd26 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8eab1d96-88bb-417a-b03e-8e2f0274e862/volumes/kubernetes.io~projected/kube-api-access-mvh42:{mountpoint:/var/lib/kubelet/pods/8eab1d96-88bb-417a-b03e-8e2f0274e862/volumes/kubernetes.io~projected/kube-api-access-mvh42 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92a4974a-3e6e-4575-b5bd-303beb3fa201/volumes/kubernetes.io~projected/kube-api-access-jp5gx:{mountpoint:/var/lib/kubelet/pods/92a4974a-3e6e-4575-b5bd-303beb3fa201/volumes/kubernetes.io~projected/kube-api-access-jp5gx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99252037-fc1d-4055-b66b-2f0e8ab910b8/volumes/kubernetes.io~projected/kube-api-access-dlvzt:{mountpoint:/var/lib/kubelet/pods/99252037-fc1d-4055-b66b-2f0e8ab910b8/volumes/kubernetes.io~projected/kube-api-access-dlvzt major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99252037-fc1d-4055-b66b-2f0e8ab910b8/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/99252037-fc1d-4055-b66b-2f0e8ab910b8/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99252037-fc1d-4055-b66b-2f0e8ab910b8/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/99252037-fc1d-4055-b66b-2f0e8ab910b8/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9af08ecc-3199-4c8b-b364-f42f4befeff1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9af08ecc-3199-4c8b-b364-f42f4befeff1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9af08ecc-3199-4c8b-b364-f42f4befeff1/volumes/kubernetes.io~projected/kube-api-access-nkrjv:{mountpoint:/var/lib/kubelet/pods/9af08ecc-3199-4c8b-b364-f42f4befeff1/volumes/kubernetes.io~projected/kube-api-access-nkrjv major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9af08ecc-3199-4c8b-b364-f42f4befeff1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9af08ecc-3199-4c8b-b364-f42f4befeff1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d45821a-5cff-41e5-b50c-90940f18d7c9/volumes/kubernetes.io~projected/kube-api-access-6wnl2:{mountpoint:/var/lib/kubelet/pods/9d45821a-5cff-41e5-b50c-90940f18d7c9/volumes/kubernetes.io~projected/kube-api-access-6wnl2 major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f16a8be-4c89-4786-a385-ff738661c9bb/volumes/kubernetes.io~projected/kube-api-access-62crk:{mountpoint:/var/lib/kubelet/pods/9f16a8be-4c89-4786-a385-ff738661c9bb/volumes/kubernetes.io~projected/kube-api-access-62crk major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f16a8be-4c89-4786-a385-ff738661c9bb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9f16a8be-4c89-4786-a385-ff738661c9bb/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6/volumes/kubernetes.io~projected/kube-api-access-7tdx8:{mountpoint:/var/lib/kubelet/pods/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6/volumes/kubernetes.io~projected/kube-api-access-7tdx8 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/adb5dbb9-f122-43fc-9ef9-867e6b2bb700/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/adb5dbb9-f122-43fc-9ef9-867e6b2bb700/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc45de8-4044-44c6-ac24-c39d8b2eece5/volumes/kubernetes.io~projected/kube-api-access-295mn:{mountpoint:/var/lib/kubelet/pods/afc45de8-4044-44c6-ac24-c39d8b2eece5/volumes/kubernetes.io~projected/kube-api-access-295mn major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afc45de8-4044-44c6-ac24-c39d8b2eece5/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/afc45de8-4044-44c6-ac24-c39d8b2eece5/volumes/kubernetes.io~secret/signing-key major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b40f9c33-c4fe-4176-9ae9-561592a770f6/volumes/kubernetes.io~projected/kube-api-access-tgvr4:{mountpoint:/var/lib/kubelet/pods/b40f9c33-c4fe-4176-9ae9-561592a770f6/volumes/kubernetes.io~projected/kube-api-access-tgvr4 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baf3e5ab-01d9-4caa-a239-b6df965e21e4/volumes/kubernetes.io~projected/kube-api-access-nqfzc:{mountpoint:/var/lib/kubelet/pods/baf3e5ab-01d9-4caa-a239-b6df965e21e4/volumes/kubernetes.io~projected/kube-api-access-nqfzc major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c097d835-0be1-4736-8dd8-98f375f02f47/volumes/kubernetes.io~projected/kube-api-access-qdl48:{mountpoint:/var/lib/kubelet/pods/c097d835-0be1-4736-8dd8-98f375f02f47/volumes/kubernetes.io~projected/kube-api-access-qdl48 major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c69ae355-a045-48d1-b7c9-881d2c4e8a1b/volumes/kubernetes.io~projected/kube-api-access-p2vtz:{mountpoint:/var/lib/kubelet/pods/c69ae355-a045-48d1-b7c9-881d2c4e8a1b/volumes/kubernetes.io~projected/kube-api-access-p2vtz major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3d84917-1245-4b8c-a887-fd2d668bcd2d/volumes/kubernetes.io~projected/kube-api-access-t9sn7:{mountpoint:/var/lib/kubelet/pods/d3d84917-1245-4b8c-a887-fd2d668bcd2d/volumes/kubernetes.io~projected/kube-api-access-t9sn7 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de0170e7-99d6-4102-9b88-13552cb6f421/volumes/kubernetes.io~projected/kube-api-access-2zptr:{mountpoint:/var/lib/kubelet/pods/de0170e7-99d6-4102-9b88-13552cb6f421/volumes/kubernetes.io~projected/kube-api-access-2zptr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8199465-5b50-41fd-ab4f-5201977736a3/volumes/kubernetes.io~projected/kube-api-access-v8kds:{mountpoint:/var/lib/kubelet/pods/f8199465-5b50-41fd-ab4f-5201977736a3/volumes/kubernetes.io~projected/kube-api-access-v8kds major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8199465-5b50-41fd-ab4f-5201977736a3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f8199465-5b50-41fd-ab4f-5201977736a3/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d83cb3ed2e45761911f2194209b34cc2255c518be9e235a9681abedb8305f9e8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/30e20decb13a19b2a22d2da68a30dbd4db85ed9d4bcc4d406192fbc3d9eb06ac/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5cfcb91669badaf5e069aa321697f4f0792d0dc034967de942f6355183a948bf/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/0a5df69afbe00581c6d0ac836e2c11c7da2267fc879d170d0c3bb6b1521311eb/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/1d5513b3658015b2ebaf3d82376caf2f5428d6a38bd3280dd9afd3bf59a009b0/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0d6a626e4aca64e5f6a6ebda2438b66b74ae6bdcc4249e0ca6c3a308438637af/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e9c63bbc78941f1a028a5c3edf6829cae505cea265bb02c6d074d02fa2046ba9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4a77339e6fd4f7f48996f4ef786e5894a42aef39620adf281ba2da8c3e8c01f1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/360aa26297a1f5a5d2d5eeb677c41d4f744cc24bbbe281cb9de03ebb0fe5605e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/200a1461fde8db2361fa004f3bddef94d57f3fa87801f35e2be75c1431e77eeb/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/142c83b250bb9f99397c4e0400fa59cc14d9960e16a4e9aa06feaded9cae9ca0/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/3ecce6de12dd72938a43086c14f35d2436962ac84ab46eda41eb84ab5bba1fee/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/afa04b0bb0312a02bdfc418fd8ad4e8bc9af4b965ab09458f7e9a67c1d331083/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/668a04cfe17bfb683086ab44644f4002f2f1c425e429c729d336e33176284ba1/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/4eece3da6e16af9442423ecc584515560eea905d814b3f9f0a7233ac4e45551e/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/08e4e810cc4a09af107af64df1fd14c882227451c515bdac6a8c4f323370c5b2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9e91fa1b80938abf4c688b94c3f99746a449ad35cad2796e56e9aacf84e3079b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ad69b25aadcc8de20eb35eef78317b798570023ca42755d835eaf52c52433e22/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2ecec573092cde1961f43b10803c2a19216cd38ae886f18db55e1cf384190f6c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7a44f09b03c237db67ffd98d1294ea57811f77b84f8bc1ec951cf7b24a753538/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/15317e20645d4519506a8971346a8329751d987b8d67d3238bbb0cb6e268c90e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/884ff4484efa64ec26673d08b4385f32e879d0c115155cffbf4eb0a75eafbd41/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7280a8e7774d096d715e7ea95193ec1239a4177695f7ff006f8ba5ace3f40320/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/b2096d5701d6a8aab711f93f101f6090b35993df635ff5f3c54d741d62fc0cdf/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/04e594520e6b0cb09be4dc926112c8c75a6d81683eb08395f7649e67e7fb9d93/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/80aec4823dda0956cfb2c46d7660ce57531d3fcabd3fc098c038e21b0e72e76f/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/81d19e9df5e627929dc7d06188becbaa5829a02643c4b86015dce7f6a73b17a8/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/1b17dcdad416473517af4c005a4d7a39220786f4cd64677ac236b37ea2e40731/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/2ca8e8cef5653b570f02b168cdf56f3a9b302c96353f9530bbb0d1b7565cc9f7/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/263287ad2544e2b143c9fad850c7ca13ae658cfe5f174bbdc98361bea4c1c892/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/bd6bb18ea27862111ba365cb4303d67822279bb92e4ff14c4e57fbfafd96f286/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/d6cefcdb39e9a9869bdbfde9c1d78dc801ee6ac139941cf7aa07fc057966295c/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/8f5a2cba1ea2427de6f8602b3a49e69feb2a1ddf87fd27a016c2082faea05da7/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/2fc803169d1f88e76ae8d4d5f1d49252ea99e140ed069cdfc2fa41798fff716e/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/24a158871f995e4795ae88ae5bfaa5e0aa8fdad93e69eaab59a41df2e9434156/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/c8d6756bbea19042adaeb327fa4aabb10758784602e8bb0a4b67d164b1549c89/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/62bbe73ba0c3ca22bddaa77a04cc076ca4059ecdb584ae911fb2c2416cce4b1a/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/d0fd873f0fb7ecb7d5097d91f8f38239fff0b1550f16d7249093a51ec9c8f610/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/af2fadcb59807d125d46b308922e13c2a6d316257510473cc4bd5f7a1a7676bd/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/42d4fd61849895d039d8ce6e14ee1d129178a14ff09931e8c90d9ff85d557bcd/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/e4eda3f6d0927d4430c989577bbc7a7a2c95c87a2b294eb4e28ee84ff5912aac/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/26c45bf87d4d589626d4e78fc5ce40e1a270f2c7431e17227e3573450625f8b4/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/a531e526b77433abf7dd6a74e91b59b84f144fa98ce2bd2be3a2c303faaf399d/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/1a484a1cf22846dc7a5ac7cdb90b1018d0daa9e74e1f606becae675369eb7510/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/18d9f8c5176123bbc1b6e0a258afd81879c3fd0be9ac69b730743ab2511c52fd/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/ff1ed115a406c67ddcda232e42fbb007f4722fc5a962584372ada88f3853b2d3/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/8ef0c35f417f28171067d476cb6cbb31e1198acbaa991b9fc95112649c17f9b9/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/022c87e967b71c7057e86c684ddea5071a1199c87a60ec71958cd42e0e27ceb6/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/a9996d06858288b7f5246f95c72d999f7ed3f2927068abf4411d78e9022f7661/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/6055968464ecbf454a01e2b61ee289761656620b868ef6f261ac93925eaba1c0/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/2d95bbb497a5093598dbc300124222a2a51e5c0ba43ff2f52b16210a41a90013/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/49e274d73415403274601e381e5da34714d14645a90941fa74f16551dbb62903/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/44bba5488d458fcc83273882ab8651f69e7567a5f45bdd0d2a41115e84d18d36/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2c1a0d56160346ca9b148f653a5853a602f4caea4835c962040fed499bf75c4c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/91a41bf0fe74bd16cd00be86aafa05ecd99c8a768c3fc2019fa4914cea3a997f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8c8b6f3a369b90c459f4023c07b4aaf316632a7a1affbc9df5224df77f0c2343/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/271a63502c457edc52439d63afaa0f842c6a5320bcd48b7b9447f2932a094649/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c2e9ffc7de6ecf45074567ea543bff610fc32eca7c3ab3cb306458fb0d533b8e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/57fc9934d997bba6859e90fb34ac727c3d68a0818c9787f2ec464f29b8f1e456/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/b5e575c3d8e7be37aecc22fd47c07738abb497f69dbb52ad6a0f0ee669c5f898/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/0c8e86d4deab908b4a055a4bf823bb61d64a32be09b51f8a4a58135aaf08eea6/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/94986f5e8b9595bb9f607ba4df5706eea563c7c981db40242113886569378ef9/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.236924    5060 manager.go:217] Machine: {Timestamp:2026-10-30 22:18:39.236467591 +0000 UTC m=+0.056205400 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3217837 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2603805aae9ef7962cc5589ad9da9c SystemUUID:ec260380-5aae-9ef7-962c-c5589ad9da9c BootID:aec00bea-8e2a-4cad-9a25-9de3830471a6 Filesystems:[{Device:/var/lib/kubelet/pods/9af08ecc-3199-4c8b-b364-f42f4befeff1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f8199465-5b50-41fd-ab4f-5201977736a3/volumes/kubernetes.io~projected/kube-api-access-v8kds DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f16a8be-4c89-4786-a385-ff738661c9bb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/012d43c7-e3f2-4931-877e-cdf66eaf61b6/volumes/kubernetes.io~projected/kube-api-access-sgv2b DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46263b706eaf4cfcc2d5ae28dacaa4a4ccd35a80ec5b8274f7f82a9e42799b0b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec63bb01333f4ac171a9de591aa0030b6a7f80fd57913288b79fe49c64bbc597/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e39b79b239aba08b4d306d91ee3e02b66c2352419b90dba87222d6a154cc462d/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/afc45de8-4044-44c6-ac24-c39d8b2eece5/volumes/kubernetes.io~projected/kube-api-access-295mn DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afa2b2cc445d9210c603695d7342906088b6a9ab69fe0776876ca02ce9505ffc/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/83a06c56-145a-4a52-baca-2d01c679e581/volumes/kubernetes.io~projected/kube-api-access-v87rd DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3d84917-1245-4b8c-a887-fd2d668bcd2d/volumes/kubernetes.io~projected/kube-api-access-t9sn7 DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8e71c29d-cd72-42ae-a0de-1500b78c89b0/volumes/kubernetes.io~projected/kube-api-access-9xd26 DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c69ae355-a045-48d1-b7c9-881d2c4e8a1b/volumes/kubernetes.io~projected/kube-api-access-p2vtz DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49a89c7c4427f3075d63943fc81dd663470b547b2471beb6f7981c7bab57c811/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6/volumes/kubernetes.io~projected/kube-api-access-7tdx8 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92a4974a-3e6e-4575-b5bd-303beb3fa201/volumes/kubernetes.io~projected/kube-api-access-jp5gx DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/88c3c753-4382-452d-898d-732766a65bf1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/188a085b-a512-440b-90ec-30acc66e4171/volumes/kubernetes.io~projected/kube-api-access-86vmt DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/adb5dbb9-f122-43fc-9ef9-867e6b2bb700/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8eab1d96-88bb-417a-b03e-8e2f0274e862/volumes/kubernetes.io~projected/kube-api-access-mvh42 DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99252037-fc1d-4055-b66b-2f0e8ab910b8/volumes/kubernetes.io~projected/kube-api-access-dlvzt DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2e346445-18a9-494c-9b3f-6c0860f631f7/volumes/kubernetes.io~projected/kube-api-access-t8m5m DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9aed098c2b53ad0567a37b484ceec5bce0bad98ee0cc8f230ce74329d7f4b3ab/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9af08ecc-3199-4c8b-b364-f42f4befeff1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e69d09c541b7f90c5b73e2effd2c7846cf2d1059e71f811d091108f27cbfe01/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/16ece3f9-76fd-4c93-b915-d1c8cec664fb/volumes/kubernetes.io~projected/kube-api-access-nq7tw DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afc45de8-4044-44c6-ac24-c39d8b2eece5/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99252037-fc1d-4055-b66b-2f0e8ab910b8/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e55f947e1675bf5ae5340da6e81b272c1df6274582811d79d0f8defc375aad86/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/064263fb24d1f1530e87ccd92388446309aac371f65d39927ceffeadc0e43185/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d739400426ebca3446563046bb9acb55606de90d38407363c8032e47bb1a87d2/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00a6f191b21baf3cff1ae517e24ae8f98dad92eda7e8c5aa5e3d1a12f5c1179e/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/407cf470-1b82-424c-a8e2-a5f061b3b84c/volumes/kubernetes.io~projected/kube-api-access-7sfkt DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53d20e4b5c3853d4a42880651f1ccd76d8168056eefce8b175a655fcbb8d6323/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de0170e7-99d6-4102-9b88-13552cb6f421/volumes/kubernetes.io~projected/kube-api-access-2zptr DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~projected/kube-api-access-rjxjz DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/baf3e5ab-01d9-4caa-a239-b6df965e21e4/volumes/kubernetes.io~projected/kube-api-access-nqfzc DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24f6e5a3ac4313c52e40175843de822b11bd947e711492b5fb031d95fab873fb/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d45821a-5cff-41e5-b50c-90940f18d7c9/volumes/kubernetes.io~projected/kube-api-access-6wnl2 DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/16e7c643-41ee-4ac5-b285-1351ae323a1d/volumes/kubernetes.io~projected/kube-api-access-mpzp6 DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cb04f796d45f2f400dfbc9c0ac25fb7aad2c369b0e8df3d6264d6d2240d0e7c/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/012d43c7-e3f2-4931-877e-cdf66eaf61b6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/188a085b-a512-440b-90ec-30acc66e4171/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c097d835-0be1-4736-8dd8-98f375f02f47/volumes/kubernetes.io~projected/kube-api-access-qdl48 DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a7d925c-af25-4c40-a280-e870b59cb772/volumes/kubernetes.io~projected/kube-api-access-gq55l DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52e1662a1fb196c8242ec74e44da518dadad25943143c13f8984fdcd7795a6d4/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0509d0cf50d58ba7f871bd87079bee6a384612bcae5b2ed7c44494859ee66310/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ea75aed4060b21a6e5b4e68c6bbd8c4c0f466d2268d1bf40d2c983630c341be/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/458275610f6e084d5e480f1ef033cbdb6418b1d08834cd383adff7c4982a0715/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92b9a4460b7b1f4186ae630e84c9a29174ad749c12ff3b9640d144dfde700fc6/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~projected/kube-api-access-xl7kc DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/012d43c7-e3f2-4931-877e-cdf66eaf61b6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/88c3c753-4382-452d-898d-732766a65bf1/volumes/kubernetes.io~projected/kube-api-access-j92gt DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc3cef33e0cca3abe20249fbbd2709a37c6793194d800f29a59f1e5955859074/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac29696864f0f5cbe7d0bd530b05d9acbe4351b370c863f7a47b0cd3e2eefc15/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8199465-5b50-41fd-ab4f-5201977736a3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46523a1fca3e6d49c6fd38149fad1450dded49837a8cd1edadaf4ab6ebbd07b8/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83f8805eb5ebe70e2d9ebecf1c77dd7eb1aae9c735e184ff97480b2e56d9ad22/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99252037-fc1d-4055-b66b-2f0e8ab910b8/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87c8cdc5-be2e-47a0-8cde-e944fda51056/volumes/kubernetes.io~projected/kube-api-access-wm75h DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9f16a8be-4c89-4786-a385-ff738661c9bb/volumes/kubernetes.io~projected/kube-api-access-62crk DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94be0e38e90de037403a5baf5200eab1e6a6d12380ad5ebd875bf62f85d7d53c/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/428997c4-b34e-4e3c-8551-c2d8510c69f4/volumes/kubernetes.io~projected/kube-api-access-kqzxg DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1889bdc0-508e-4f97-80cb-4a5aa6e4f67f/volumes/kubernetes.io~projected/kube-api-access-9tndn DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b40f9c33-c4fe-4176-9ae9-561592a770f6/volumes/kubernetes.io~projected/kube-api-access-tgvr4 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/53de1b04-dea0-4a6f-9609-760792cc0406/volumes/kubernetes.io~projected/kube-api-access-f2x9v DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ebc14ac21416b778201bd1361cb164993cb192ee203cbc22c7419765d5202ce/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60457f9c-65b0-4547-9012-2fc4efe2b9e8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9af08ecc-3199-4c8b-b364-f42f4befeff1/volumes/kubernetes.io~projected/kube-api-access-nkrjv DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6afb2c7f1be5f304d205eeb403dad888f0f128ab0260e09b4e54435c893ee416/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16ece3f9-76fd-4c93-b915-d1c8cec664fb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00a6f191b21baf3 MacAddress:52:cc:c8:ce:11:c5 Speed:10000 Mtu:8901} {Name:064263fb24d1f15 MacAddress:72:a1:67:fe:4c:b6 Speed:10000 Mtu:8901} {Name:24f6e5a3ac4313c MacAddress:ce:38:2b:1b:bc:33 Speed:10000 Mtu:8901} {Name:458275610f6e084 MacAddress:52:43:84:f6:a5:29 Speed:10000 Mtu:8901} {Name:46523a1fca3e6d4 MacAddress:fa:e9:48:05:26:dd Speed:10000 Mtu:8901} {Name:49a89c7c4427f30 MacAddress:6e:47:cb:92:57:88 Speed:10000 Mtu:8901} {Name:52e1662a1fb196c MacAddress:06:1e:52:11:24:c5 Speed:10000 Mtu:8901} {Name:53d20e4b5c3853d MacAddress:96:c6:9d:91:9c:aa Speed:10000 Mtu:8901} {Name:5cb04f796d45f2f MacAddress:46:67:8b:d6:47:73 Speed:10000 Mtu:8901} {Name:92b9a4460b7b1f4 MacAddress:86:1b:67:95:04:f8 Speed:10000 Mtu:8901} {Name:9aed098c2b53ad0 MacAddress:c2:c9:7e:66:9e:6a Speed:10000 Mtu:8901} {Name:ac29696864f0f5c MacAddress:0a:a6:3f:ea:a8:0b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f2:b7:a2:1d Speed:0 Mtu:9001} {Name:br-int MacAddress:5e:53:86:e6:4b:40 Speed:0 Mtu:8901} {Name:d739400426ebca3 MacAddress:92:ed:17:c1:8e:06 Speed:10000 Mtu:8901} {Name:e39b79b239aba08 MacAddress:ce:53:60:41:2a:c2 Speed:10000 Mtu:8901} {Name:e55f947e1675bf5 MacAddress:56:1d:71:5c:66:17 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f2:b7:a2:1d Speed:0 Mtu:9001} {Name:fc3cef33e0cca3a MacAddress:f6:55:32:74:dd:a6 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:66:fa:af:75:98:10 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:73:7a:f6:6d:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.237265    5060 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.237307    5060 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.237620    5060 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.237645    5060 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-251.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.237789    5060 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.237797    5060 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.237807    5060 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.237820    5060 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.237839    5060 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.237993    5060 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.238261    5060 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.238318    5060 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.238332    5060 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.238368    5060 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.238381    5060 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.238394    5060 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.239603    5060 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.239622    5060 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.240380    5060 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.242735    5060 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.242746    5060 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.242848    5060 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243030    5060 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243057    5060 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243138    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243151    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243159    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243168    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243176    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243190    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243198    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243203    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243210    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243218    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243230    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243240    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243265    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:18:39.243757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.243273    5060 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:18:39.246769 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.246753    5060 server.go:1263] "Started kubelet"
Oct 30 22:18:39.246872 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.246824    5060 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:18:39.247481 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.246845    5060 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:18:39.247481 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.246995    5060 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:18:39.247481 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.247431    5060 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:18:39.247294 ip-10-0-138-251 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:18:39.248448 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.248433    5060 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:18:39.252686 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.251880    5060 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:18:39.252686 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.252011    5060 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:18:39.252686 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.252023    5060 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:18:39.252686 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.251880    5060 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:18:39.252686 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.252179    5060 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:18:39.253618 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.253047    5060 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:13:00 +0000 UTC" deadline="2028-05-20 08:09:42.136930267 +0000 UTC"
Oct 30 22:18:39.253618 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.253089    5060 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13617h51m2.883845742s"
Oct 30 22:18:39.253618 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:39.252086    5060 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-251.us-east-2.compute.internal\" not found"
Oct 30 22:18:39.253784 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.253726    5060 factory.go:55] Registering systemd factory
Oct 30 22:18:39.253784 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.253744    5060 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:18:39.254872 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.254850    5060 factory.go:153] Registering CRI-O factory
Oct 30 22:18:39.254872 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.254867    5060 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:18:39.255015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.254923    5060 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:18:39.255015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.254943    5060 factory.go:103] Registering Raw factory
Oct 30 22:18:39.255015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.254959    5060 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:18:39.255322 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.255308    5060 manager.go:319] Starting recovery of all containers
Oct 30 22:18:39.256960 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.256931    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555a97b6-857c-4374-8939-1eed2d487b4c" volumeName="kubernetes.io/empty-dir/555a97b6-857c-4374-8939-1eed2d487b4c-ca-trust-extracted" seLinuxMountContext=""
Oct 30 22:18:39.257125 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.256964    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555a97b6-857c-4374-8939-1eed2d487b4c" volumeName="kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 22:18:39.257125 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.256989    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555a97b6-857c-4374-8939-1eed2d487b4c" volumeName="kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-installation-pull-secrets" seLinuxMountContext=""
Oct 30 22:18:39.257246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257186    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92a4974a-3e6e-4575-b5bd-303beb3fa201" volumeName="kubernetes.io/empty-dir/92a4974a-3e6e-4575-b5bd-303beb3fa201-tmp-dir" seLinuxMountContext=""
Oct 30 22:18:39.257246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257212    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6" volumeName="kubernetes.io/configmap/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-serviceca" seLinuxMountContext=""
Oct 30 22:18:39.257246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257222    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d45821a-5cff-41e5-b50c-90940f18d7c9" volumeName="kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 22:18:39.257246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257232    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc45de8-4044-44c6-ac24-c39d8b2eece5" volumeName="kubernetes.io/configmap/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-cabundle" seLinuxMountContext=""
Oct 30 22:18:39.257246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257245    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8199465-5b50-41fd-ab4f-5201977736a3" volumeName="kubernetes.io/projected/f8199465-5b50-41fd-ab4f-5201977736a3-kube-api-access-v8kds" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257256    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16e7c643-41ee-4ac5-b285-1351ae323a1d" volumeName="kubernetes.io/projected/16e7c643-41ee-4ac5-b285-1351ae323a1d-kube-api-access-mpzp6" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257269    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16ece3f9-76fd-4c93-b915-d1c8cec664fb" volumeName="kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-service-ca-bundle" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257277    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" volumeName="kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-trusted-ca" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257288    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" volumeName="kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-bound-sa-token" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257296    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3d84917-1245-4b8c-a887-fd2d668bcd2d" volumeName="kubernetes.io/empty-dir/d3d84917-1245-4b8c-a887-fd2d668bcd2d-data-volume" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257304    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60457f9c-65b0-4547-9012-2fc4efe2b9e8" volumeName="kubernetes.io/configmap/60457f9c-65b0-4547-9012-2fc4efe2b9e8-konnectivity-ca" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257319    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c3c753-4382-452d-898d-732766a65bf1" volumeName="kubernetes.io/projected/88c3c753-4382-452d-898d-732766a65bf1-kube-api-access-j92gt" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257329    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8eab1d96-88bb-417a-b03e-8e2f0274e862" volumeName="kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257346    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99252037-fc1d-4055-b66b-2f0e8ab910b8" volumeName="kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-default-certificate" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257355    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d45821a-5cff-41e5-b50c-90940f18d7c9" volumeName="kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-service-ca-bundle" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257365    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d45821a-5cff-41e5-b50c-90940f18d7c9" volumeName="kubernetes.io/empty-dir/9d45821a-5cff-41e5-b50c-90940f18d7c9-archives-path" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257373    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c097d835-0be1-4736-8dd8-98f375f02f47" volumeName="kubernetes.io/projected/c097d835-0be1-4736-8dd8-98f375f02f47-kube-api-access-qdl48" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257381    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16ece3f9-76fd-4c93-b915-d1c8cec664fb" volumeName="kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257393    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" volumeName="kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257402    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e71c29d-cd72-42ae-a0de-1500b78c89b0" volumeName="kubernetes.io/configmap/8e71c29d-cd72-42ae-a0de-1500b78c89b0-config-volume" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257412    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc45de8-4044-44c6-ac24-c39d8b2eece5" volumeName="kubernetes.io/projected/afc45de8-4044-44c6-ac24-c39d8b2eece5-kube-api-access-295mn" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257420    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afc45de8-4044-44c6-ac24-c39d8b2eece5" volumeName="kubernetes.io/secret/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-key" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257428    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de0170e7-99d6-4102-9b88-13552cb6f421" volumeName="kubernetes.io/projected/de0170e7-99d6-4102-9b88-13552cb6f421-kube-api-access-2zptr" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257438    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e346445-18a9-494c-9b3f-6c0860f631f7" volumeName="kubernetes.io/projected/2e346445-18a9-494c-9b3f-6c0860f631f7-kube-api-access-t8m5m" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257447    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a7d925c-af25-4c40-a280-e870b59cb772" volumeName="kubernetes.io/projected/3a7d925c-af25-4c40-a280-e870b59cb772-kube-api-access-gq55l" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257462    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="428997c4-b34e-4e3c-8551-c2d8510c69f4" volumeName="kubernetes.io/projected/428997c4-b34e-4e3c-8551-c2d8510c69f4-kube-api-access-kqzxg" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257473    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="012d43c7-e3f2-4931-877e-cdf66eaf61b6" volumeName="kubernetes.io/projected/012d43c7-e3f2-4931-877e-cdf66eaf61b6-kube-api-access-sgv2b" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257481    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16ece3f9-76fd-4c93-b915-d1c8cec664fb" volumeName="kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-tmp" seLinuxMountContext=""
Oct 30 22:18:39.257473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257489    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" volumeName="kubernetes.io/empty-dir/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-ca-trust-extracted" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257555    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="407cf470-1b82-424c-a8e2-a5f061b3b84c" volumeName="kubernetes.io/projected/407cf470-1b82-424c-a8e2-a5f061b3b84c-kube-api-access-7sfkt" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257588    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555a97b6-857c-4374-8939-1eed2d487b4c" volumeName="kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-kube-api-access-xl7kc" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257617    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87c8cdc5-be2e-47a0-8cde-e944fda51056" volumeName="kubernetes.io/projected/87c8cdc5-be2e-47a0-8cde-e944fda51056-kube-api-access-wm75h" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257640    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c3c753-4382-452d-898d-732766a65bf1" volumeName="kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-config" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257653    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e71c29d-cd72-42ae-a0de-1500b78c89b0" volumeName="kubernetes.io/projected/8e71c29d-cd72-42ae-a0de-1500b78c89b0-kube-api-access-9xd26" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257666    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af08ecc-3199-4c8b-b364-f42f4befeff1" volumeName="kubernetes.io/projected/9af08ecc-3199-4c8b-b364-f42f4befeff1-kube-api-access-nkrjv" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257684    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af08ecc-3199-4c8b-b364-f42f4befeff1" volumeName="kubernetes.io/secret/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257696    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d45821a-5cff-41e5-b50c-90940f18d7c9" volumeName="kubernetes.io/projected/9d45821a-5cff-41e5-b50c-90940f18d7c9-kube-api-access-6wnl2" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257885    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f16a8be-4c89-4786-a385-ff738661c9bb" volumeName="kubernetes.io/projected/9f16a8be-4c89-4786-a385-ff738661c9bb-kube-api-access-62crk" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257900    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f16a8be-4c89-4786-a385-ff738661c9bb" volumeName="kubernetes.io/secret/9f16a8be-4c89-4786-a385-ff738661c9bb-serving-cert" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257913    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b40f9c33-c4fe-4176-9ae9-561592a770f6" volumeName="kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257922    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16ece3f9-76fd-4c93-b915-d1c8cec664fb" volumeName="kubernetes.io/projected/16ece3f9-76fd-4c93-b915-d1c8cec664fb-kube-api-access-nq7tw" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257929    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555a97b6-857c-4374-8939-1eed2d487b4c" volumeName="kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-trusted-ca" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257940    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c3c753-4382-452d-898d-732766a65bf1" volumeName="kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-trusted-ca" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257948    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99252037-fc1d-4055-b66b-2f0e8ab910b8" volumeName="kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-stats-auth" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257960    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af08ecc-3199-4c8b-b364-f42f4befeff1" volumeName="kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257969    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6" volumeName="kubernetes.io/projected/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-kube-api-access-7tdx8" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257978    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c69ae355-a045-48d1-b7c9-881d2c4e8a1b" volumeName="kubernetes.io/configmap/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-telemetry-config" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257989    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="012d43c7-e3f2-4931-877e-cdf66eaf61b6" volumeName="kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-tuned" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.257999    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af08ecc-3199-4c8b-b364-f42f4befeff1" volumeName="kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-env-overrides" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258015    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53de1b04-dea0-4a6f-9609-760792cc0406" volumeName="kubernetes.io/projected/53de1b04-dea0-4a6f-9609-760792cc0406-kube-api-access-f2x9v" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258028    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a06c56-145a-4a52-baca-2d01c679e581" volumeName="kubernetes.io/configmap/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-config" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258045    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e71c29d-cd72-42ae-a0de-1500b78c89b0" volumeName="kubernetes.io/empty-dir/8e71c29d-cd72-42ae-a0de-1500b78c89b0-tmp-dir" seLinuxMountContext=""
Oct 30 22:18:39.258592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258559    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="adb5dbb9-f122-43fc-9ef9-867e6b2bb700" volumeName="kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258632    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="012d43c7-e3f2-4931-877e-cdf66eaf61b6" volumeName="kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-tmp" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258653    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16ece3f9-76fd-4c93-b915-d1c8cec664fb" volumeName="kubernetes.io/secret/16ece3f9-76fd-4c93-b915-d1c8cec664fb-serving-cert" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258666    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="188a085b-a512-440b-90ec-30acc66e4171" volumeName="kubernetes.io/secret/188a085b-a512-440b-90ec-30acc66e4171-cert" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258680    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a06c56-145a-4a52-baca-2d01c679e581" volumeName="kubernetes.io/projected/83a06c56-145a-4a52-baca-2d01c679e581-kube-api-access-v87rd" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258711    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87c8cdc5-be2e-47a0-8cde-e944fda51056" volumeName="kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258723    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88c3c753-4382-452d-898d-732766a65bf1" volumeName="kubernetes.io/secret/88c3c753-4382-452d-898d-732766a65bf1-serving-cert" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258739    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99252037-fc1d-4055-b66b-2f0e8ab910b8" volumeName="kubernetes.io/projected/99252037-fc1d-4055-b66b-2f0e8ab910b8-kube-api-access-dlvzt" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258764    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f16a8be-4c89-4786-a385-ff738661c9bb" volumeName="kubernetes.io/configmap/9f16a8be-4c89-4786-a385-ff738661c9bb-config" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258780    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="188a085b-a512-440b-90ec-30acc66e4171" volumeName="kubernetes.io/projected/188a085b-a512-440b-90ec-30acc66e4171-kube-api-access-86vmt" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258797    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87c8cdc5-be2e-47a0-8cde-e944fda51056" volumeName="kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-daemon-config" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258808    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b40f9c33-c4fe-4176-9ae9-561592a770f6" volumeName="kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258825    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baf3e5ab-01d9-4caa-a239-b6df965e21e4" volumeName="kubernetes.io/projected/baf3e5ab-01d9-4caa-a239-b6df965e21e4-kube-api-access-nqfzc" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258837    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3d84917-1245-4b8c-a887-fd2d668bcd2d" volumeName="kubernetes.io/projected/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-api-access-t9sn7" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258849    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8199465-5b50-41fd-ab4f-5201977736a3" volumeName="kubernetes.io/configmap/f8199465-5b50-41fd-ab4f-5201977736a3-config" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258864    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8199465-5b50-41fd-ab4f-5201977736a3" volumeName="kubernetes.io/secret/f8199465-5b50-41fd-ab4f-5201977736a3-serving-cert" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258877    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16ece3f9-76fd-4c93-b915-d1c8cec664fb" volumeName="kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-snapshots" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258895    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" volumeName="kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-certificates" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258907    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" volumeName="kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-installation-pull-secrets" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258920    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555a97b6-857c-4374-8939-1eed2d487b4c" volumeName="kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-bound-sa-token" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258936    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60457f9c-65b0-4547-9012-2fc4efe2b9e8" volumeName="kubernetes.io/secret/60457f9c-65b0-4547-9012-2fc4efe2b9e8-agent-certs" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258949    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af08ecc-3199-4c8b-b364-f42f4befeff1" volumeName="kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-config" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258967    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88d59947-036f-412d-9ee6-f7565b0d4d8b" volumeName="kubernetes.io/configmap/88d59947-036f-412d-9ee6-f7565b0d4d8b-nginx-conf" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258979    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b40f9c33-c4fe-4176-9ae9-561592a770f6" volumeName="kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.258992    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="555a97b6-857c-4374-8939-1eed2d487b4c" volumeName="kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-registry-certificates" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.259008    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92a4974a-3e6e-4575-b5bd-303beb3fa201" volumeName="kubernetes.io/projected/92a4974a-3e6e-4575-b5bd-303beb3fa201-kube-api-access-jp5gx" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.259023    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b40f9c33-c4fe-4176-9ae9-561592a770f6" volumeName="kubernetes.io/projected/b40f9c33-c4fe-4176-9ae9-561592a770f6-kube-api-access-tgvr4" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.259040    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1889bdc0-508e-4f97-80cb-4a5aa6e4f67f" volumeName="kubernetes.io/projected/1889bdc0-508e-4f97-80cb-4a5aa6e4f67f-kube-api-access-9tndn" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.259053    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="407cf470-1b82-424c-a8e2-a5f061b3b84c" volumeName="kubernetes.io/configmap/407cf470-1b82-424c-a8e2-a5f061b3b84c-iptables-alerter-script" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.259066    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c69ae355-a045-48d1-b7c9-881d2c4e8a1b" volumeName="kubernetes.io/projected/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-kube-api-access-p2vtz" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.259083    5060 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" volumeName="kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-kube-api-access-rjxjz" seLinuxMountContext=""
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.259094    5060 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:18:39.259602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.259102    5060 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:18:39.263578 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.262974    5060 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:18:39.268003 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.267786    5060 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:18:39.269664 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.269645    5060 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:18:39.270449 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.270429    5060 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 22:18:39.280711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.280692    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/1.log"
Oct 30 22:18:39.281049 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.281027    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/1.log"
Oct 30 22:18:39.281611 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.281573    5060 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:18:39.283061 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.283042    5060 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:18:39.283146 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.283067    5060 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:18:39.283146 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.283092    5060 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:18:39.283146 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:39.283131    5060 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 22:18:39.287193 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.287168    5060 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:18:39.305172 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.305130    5060 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="81b5c6a914092fc16adfff1186008ec03f8e7b9d64448acc5a6b0555ff903d48" exitCode=0
Oct 30 22:18:39.305172 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.305172    5060 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="d508648ad38bb794d6872a25e1424eea0727b1b88134e91d31ecb23617ec0b2d" exitCode=0
Oct 30 22:18:39.305172 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.305179    5060 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="4c62e21dfdb661764b385623f9297abcc452aafed2dcf7fe10400cee140fffb3" exitCode=0
Oct 30 22:18:39.305368 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.305186    5060 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="018f861db55ca7375d1476023c7c1437a079cd134a5df91a9e67f3f678ab0e92" exitCode=0
Oct 30 22:18:39.305368 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.305193    5060 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="c8500d159b7b748246b129391cb66578b407c042f241c861acfa2a9dff08da6a" exitCode=0
Oct 30 22:18:39.305368 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.305198    5060 generic.go:356] "Generic (PLEG): container finished" podID="b40f9c33-c4fe-4176-9ae9-561592a770f6" containerID="4c34b2608bfaf95f34f80ee411671799b36815c4699aa7384fcccce776a80ccf" exitCode=0
Oct 30 22:18:39.312605 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.312580    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/1.log"
Oct 30 22:18:39.312708 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.312612    5060 generic.go:356] "Generic (PLEG): container finished" podID="88c3c753-4382-452d-898d-732766a65bf1" containerID="6b06fc816b5d2acc17993f19f7a4cabafe3351bda1a804c24f06f570d748d8af" exitCode=255
Oct 30 22:18:39.323545 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.323500    5060 generic.go:356] "Generic (PLEG): container finished" podID="65deeb7675a6280ac03893337db51c32" containerID="3642d671d5430b07486cc8c49fec1a1474b3d96262c1fd6903f545a48a5e8cea" exitCode=0
Oct 30 22:18:39.354352 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.354290    5060 manager.go:324] Recovery completed
Oct 30 22:18:39.377261 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377237    5060 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:18:39.377261 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377254    5060 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:18:39.377432 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377272    5060 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:18:39.377432 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377412    5060 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 22:18:39.377489 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377423    5060 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 22:18:39.377489 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377448    5060 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 22:18:39.377489 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377454    5060 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 22:18:39.377489 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377459    5060 policy_none.go:50] "Start"
Oct 30 22:18:39.377489 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377471    5060 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:18:39.377489 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377480    5060 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:18:39.377658 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377619    5060 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 22:18:39.377658 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377626    5060 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 22:18:39.377658 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.377631    5060 policy_none.go:44] "Start"
Oct 30 22:18:39.382048 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.382018    5060 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:18:39.383855 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:39.383840    5060 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 22:18:39.384618 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.384606    5060 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:18:39.385058 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:39.384632    5060 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:18:39.385107 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.385060    5060 server.go:87] "Starting device plugin registration server"
Oct 30 22:18:39.385304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.385295    5060 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:18:39.385336 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.385308    5060 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:18:39.385424 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.385411    5060 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:18:39.386570 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.386526    5060 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:18:39.386653 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.386581    5060 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:18:39.386653 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.386589    5060 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:18:39.386724 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.386648    5060 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:18:39.386565298Z","UUID":"0065aaba-ce30-4112-9a8f-fa468d996e84","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:18:39.387142 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:39.387125    5060 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:18:39.388125 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.388098    5060 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:18:39.388218 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.388138    5060 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:18:39.485993 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.485945    5060 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:18:39.487360 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.487337    5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 22:18:39.487458 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.487384    5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 22:18:39.487458 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.487399    5060 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-251.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 22:18:39.487458 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.487440    5060 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.509669 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.509649    5060 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.509762 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.509734    5060 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.584834 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.584785    5060 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"]
Oct 30 22:18:39.585458 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.585420    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" event={"ID":"a5c727ac326546386befce51c4d99c6d","Type":"ContainerStarted","Data":"605458c066ca7a377d2619a0af91912e10f5c6825d4bb0948428c733d5cdb681"}
Oct 30 22:18:39.585493 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.585468    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" event={"ID":"a5c727ac326546386befce51c4d99c6d","Type":"ContainerStarted","Data":"94be0e38e90de037403a5baf5200eab1e6a6d12380ad5ebd875bf62f85d7d53c"}
Oct 30 22:18:39.585493 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.585482    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.585569 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.585540    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal" event={"ID":"65deeb7675a6280ac03893337db51c32","Type":"ContainerStarted","Data":"049607b4cfec0b036c6396af3a97baccd7224a0379d2a97d0be157456f3c6717"}
Oct 30 22:18:39.585569 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.585562    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal" event={"ID":"65deeb7675a6280ac03893337db51c32","Type":"ContainerDied","Data":"3642d671d5430b07486cc8c49fec1a1474b3d96262c1fd6903f545a48a5e8cea"}
Oct 30 22:18:39.585641 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.585579    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal" event={"ID":"65deeb7675a6280ac03893337db51c32","Type":"ContainerStarted","Data":"3ebc14ac21416b778201bd1361cb164993cb192ee203cbc22c7419765d5202ce"}
Oct 30 22:18:39.585736 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.585724    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.606289 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.606239    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:18:39.606371 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:39.606286    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.606371 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.606331    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:18:39.606371 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:39.606363    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.672661 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.672635    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.672661 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.672666    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.672801 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.672685    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5c727ac326546386befce51c4d99c6d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"a5c727ac326546386befce51c4d99c6d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.773226 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.773193    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.773226 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.773226    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.773374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.773286    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.773374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.773297    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65deeb7675a6280ac03893337db51c32-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"65deeb7675a6280ac03893337db51c32\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.773374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.773300    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5c727ac326546386befce51c4d99c6d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"a5c727ac326546386befce51c4d99c6d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:39.773374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:39.773329    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5c727ac326546386befce51c4d99c6d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal\" (UID: \"a5c727ac326546386befce51c4d99c6d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:40.239739 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.239708    5060 apiserver.go:52] "Watching apiserver"
Oct 30 22:18:40.261694 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.261670    5060 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:18:40.262582 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.262554    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b","openshift-cluster-node-tuning-operator/tuned-ggnd6","kube-system/kas-connection-checker-685895c7c9-tsv8h","openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6","openshift-insights/insights-runtime-extractor-6mwl7","openshift-network-diagnostics/network-check-target-tgn5k","kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal","openshift-ingress-canary/ingress-canary-rvz8k","openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j","openshift-multus/network-metrics-daemon-hmgsj","openshift-network-operator/iptables-alerter-vhkmj","openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h","kube-system/konnectivity-agent-wb9bv","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal","openshift-dns/node-resolver-qqwss","openshift-ingress/router-default-6668ff59b7-vglrn","openshift-service-ca/service-ca-5fffc97bdf-qlkzv","openshift-image-registry/image-registry-5bfb9545c9-tkcrm","openshift-insights/periodic-gathering-qstj7-lntsk","openshift-console-operator/console-operator-7cd56b8dbc-dd7n4","openshift-image-registry/node-ca-7cjdl","openshift-image-registry/image-registry-5559797b78-pnvl9","kube-system/global-pull-secret-syncer-xkt2l","openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd","openshift-multus/multus-additional-cni-plugins-64cvm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl","kube-system/kas-connection-checker-685895c7c9-b4mzj","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws","openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k","openshift-multus/multus-kfshl","kube-system/kas-connection-checker-685895c7c9-8cbpq","openshift-dns/dns-default-6wrrt","openshift-insights/insights-operator-7c4986d449-xkcch","openshift-ovn-kubernetes/ovnkube-node-7ppff","openshift-network-console/networking-console-plugin-86c478bb88-grpzl"]
Oct 30 22:18:40.266820 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.266796    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:40.267135 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.267108    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.267347 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.267318    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:40.267426 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.267404    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.268208 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.268194    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.269337 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.269321    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.270078 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.270062    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:40.270563 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.270547    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:40.272871 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.272849    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 22:18:40.273010 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.272994    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.273204 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273186    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 22:18:40.273327 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273304    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.273417 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273351    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.273417 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273370    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 22:18:40.273528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273429    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 22:18:40.273528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273433    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 22:18:40.273528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273457    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 22:18:40.273528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273468    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.273528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273475    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.273528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273479    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 22:18:40.273528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273468    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.273528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273519    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 22:18:40.273528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273526    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 22:18:40.273528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273531    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.273910 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273565    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 22:18:40.273910 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273578    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.273910 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273647    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.273910 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273660    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.273910 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273708    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.273910 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273711    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.273910 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.273767    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 22:18:40.276898 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.276880    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 22:18:40.276898 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.276878    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 22:18:40.277032 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.277020    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.277286 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.277264    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 22:18:40.277949 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.277925    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kkwnr\""
Oct 30 22:18:40.278020 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.277275    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:18:40.278070 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.277964    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 22:18:40.278431 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.278398    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 22:18:40.278500 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.278441    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sdfwl\""
Oct 30 22:18:40.278582 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.278541    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:18:40.278788 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.278772    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jmj7l\""
Oct 30 22:18:40.279082 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.279064    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:18:40.279480 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.279456    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ms5nl\""
Oct 30 22:18:40.279576 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.279499    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-szxgd\""
Oct 30 22:18:40.279728 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.279698    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-glgqx\""
Oct 30 22:18:40.280347 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.280328    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:18:40.281019 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.280996    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 22:18:40.281420 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.281401    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 22:18:40.281725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.281703    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-p44cw\""
Oct 30 22:18:40.281891 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.281407    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tp94j\""
Oct 30 22:18:40.281891 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.281844    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jb8sb\""
Oct 30 22:18:40.282070 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.282043    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.282142 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.282112    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.282212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.282154    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.282212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.282200    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 22:18:40.282390 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.282375    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.282685 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.282648    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:18:40.283135 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.283116    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.283302 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.283271    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 22:18:40.283383 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.283306    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:18:40.283688 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.283675    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.283753 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.283699    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.283910 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.283895    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 22:18:40.283989 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.283922    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 22:18:40.283989 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.283986    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.284091 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284045    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.284091 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284050    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.284091 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284086    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-r4dw4\""
Oct 30 22:18:40.284212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284085    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 22:18:40.284212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284107    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.284212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284116    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:18:40.284212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284128    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.284212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284108    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.284212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284141    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:18:40.284212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284177    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.284212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284196    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284220    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284101    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284232    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284261    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284270    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284296    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284303    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jm6gb\""
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284489    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284570    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284579    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.284725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.284674    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:18:40.285316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.285299    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 22:18:40.285736 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.285721    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.286859 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.286838    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.288337 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.288322    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:18:40.288624 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.288608    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.288680 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.288645    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.288680 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.288668    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.288836 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.288824    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:18:40.290695 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.290680    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:18:40.290751 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.290739    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.290804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.290754    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 22:18:40.291021 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.291009    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.291067 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.291018    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 22:18:40.291067 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.291029    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 22:18:40.342674 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.342640    5060 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:40.354715 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.354688    5060 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:18:40.357343 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.357328    5060 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:18:40.357423 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.357369    5060 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 22:18:40.374057 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374019    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:18:40.374295 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374270    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4x7f7\""
Oct 30 22:18:40.374371 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374295    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vjg9f\""
Oct 30 22:18:40.374371 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374296    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jfcbv\""
Oct 30 22:18:40.374536 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374493    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 22:18:40.374648 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374635    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:18:40.374709 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374671    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:18:40.374761 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374733    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9jqsq\""
Oct 30 22:18:40.374761 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374737    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qwcf4\""
Oct 30 22:18:40.374830 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374748    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-srblb\""
Oct 30 22:18:40.374830 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374764    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9pq22\""
Oct 30 22:18:40.374929 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374839    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-snj5d\""
Oct 30 22:18:40.374929 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.374901    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dhvzl\""
Oct 30 22:18:40.375951 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.375933    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-tmp\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.375996 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.375967    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqfzc\" (UniqueName: \"kubernetes.io/projected/baf3e5ab-01d9-4caa-a239-b6df965e21e4-kube-api-access-nqfzc\") pod \"kas-connection-checker-685895c7c9-b4mzj\" (UID: \"baf3e5ab-01d9-4caa-a239-b6df965e21e4\") " pod="kube-system/kas-connection-checker-685895c7c9-b4mzj"
Oct 30 22:18:40.376051 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376000    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq7tw\" (UniqueName: \"kubernetes.io/projected/16ece3f9-76fd-4c93-b915-d1c8cec664fb-kube-api-access-nq7tw\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.376051 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376029    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-kubernetes\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.376157 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376053    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-var-lib-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.376157 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376076    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-etc-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.376157 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376082    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-tmp\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.376157 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376099    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-cnibin\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.376157 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376122    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/407cf470-1b82-424c-a8e2-a5f061b3b84c-host-slash\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:40.376317 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376171    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-conf-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.376317 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376204    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-trusted-ca\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.376417 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376405    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.376455 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376427    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-service-ca-bundle\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:40.376455 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376446    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-ca-trust-extracted\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.376533 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376470    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-kubelet\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.376533 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376494    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3d84917-1245-4b8c-a887-fd2d668bcd2d-crio-socket\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.376618 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376546    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:40.376618 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376554    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-ca-trust-extracted\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.376618 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376592    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-service-ca-bundle\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.376730 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376641    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-service-ca-bundle\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:40.376836 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376820    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-service-ca-bundle\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.376874 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376851    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:40.376907 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376883    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8m5m\" (UniqueName: \"kubernetes.io/projected/2e346445-18a9-494c-9b3f-6c0860f631f7-kube-api-access-t8m5m\") pod \"kas-connection-checker-685895c7c9-8cbpq\" (UID: \"2e346445-18a9-494c-9b3f-6c0860f631f7\") " pod="kube-system/kas-connection-checker-685895c7c9-8cbpq"
Oct 30 22:18:40.376948 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376908    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60457f9c-65b0-4547-9012-2fc4efe2b9e8-agent-certs\") pod \"konnectivity-agent-wb9bv\" (UID: \"60457f9c-65b0-4547-9012-2fc4efe2b9e8\") " pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:40.376948 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376930    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-node-log\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.377006 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376954    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-bin\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.377006 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.376976    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3d84917-1245-4b8c-a887-fd2d668bcd2d-data-volume\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.377074 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377011    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.377074 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377048    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-trusted-ca\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.377074 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377045    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-image-registry-private-configuration\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.377172 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377084    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3d84917-1245-4b8c-a887-fd2d668bcd2d-data-volume\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.377172 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377090    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.377265 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377173    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:40.377265 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377241    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-registration-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.377357 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377285    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-netns\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.377357 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377290    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-image-registry-private-configuration\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.377357 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377303    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ece3f9-76fd-4c93-b915-d1c8cec664fb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.377357 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377324    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-systemd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.377357 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377344    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60457f9c-65b0-4547-9012-2fc4efe2b9e8-agent-certs\") pod \"konnectivity-agent-wb9bv\" (UID: \"60457f9c-65b0-4547-9012-2fc4efe2b9e8\") " pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:40.377357 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377354    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrjv\" (UniqueName: \"kubernetes.io/projected/9af08ecc-3199-4c8b-b364-f42f4befeff1-kube-api-access-nkrjv\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.377640 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377395    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sfkt\" (UniqueName: \"kubernetes.io/projected/407cf470-1b82-424c-a8e2-a5f061b3b84c-kube-api-access-7sfkt\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:40.377640 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377420    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.377918 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377896    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:40.377986 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377490    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e71c29d-cd72-42ae-a0de-1500b78c89b0-config-volume\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:40.378038 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377982    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e71c29d-cd72-42ae-a0de-1500b78c89b0-config-volume\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:40.378038 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.377977    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kds\" (UniqueName: \"kubernetes.io/projected/f8199465-5b50-41fd-ab4f-5201977736a3-kube-api-access-v8kds\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:40.378121 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378056    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vmt\" (UniqueName: \"kubernetes.io/projected/188a085b-a512-440b-90ec-30acc66e4171-kube-api-access-86vmt\") pod \"ingress-canary-rvz8k\" (UID: \"188a085b-a512-440b-90ec-30acc66e4171\") " pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:40.378121 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378093    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.378424 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378400    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wnl2\" (UniqueName: \"kubernetes.io/projected/9d45821a-5cff-41e5-b50c-90940f18d7c9-kube-api-access-6wnl2\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:40.378588 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378566    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-lib-modules\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.378669 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378605    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-env-overrides\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.378669 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378642    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-script-lib\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.378768 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378676    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7kc\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-kube-api-access-xl7kc\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.378768 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378714    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-multus-certs\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.378856 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378767    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-cabundle\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:40.378856 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378797    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62crk\" (UniqueName: \"kubernetes.io/projected/9f16a8be-4c89-4786-a385-ff738661c9bb-kube-api-access-62crk\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:40.378856 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378829    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-etc-selinux\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.378966 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378858    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-ovn\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.378966 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378890    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdl48\" (UniqueName: \"kubernetes.io/projected/c097d835-0be1-4736-8dd8-98f375f02f47-kube-api-access-qdl48\") pod \"migrator-66df5d496b-vfc6j\" (UID: \"c097d835-0be1-4736-8dd8-98f375f02f47\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j"
Oct 30 22:18:40.378966 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378925    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.378966 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.378951    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm75h\" (UniqueName: \"kubernetes.io/projected/87c8cdc5-be2e-47a0-8cde-e944fda51056-kube-api-access-wm75h\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.379097 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379004    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-env-overrides\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.379321 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379293    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpzp6\" (UniqueName: \"kubernetes.io/projected/16e7c643-41ee-4ac5-b285-1351ae323a1d-kube-api-access-mpzp6\") pod \"migrator-66df5d496b-ds7bd\" (UID: \"16e7c643-41ee-4ac5-b285-1351ae323a1d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd"
Oct 30 22:18:40.379406 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379367    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-config\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.379460 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379398    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.379460 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379400    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-script-lib\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.379460 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379420    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/555a97b6-857c-4374-8939-1eed2d487b4c-ca-trust-extracted\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.379642 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379462    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-var-lib-kubelet\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.379642 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379491    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-systemd-units\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.379642 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379535    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/555a97b6-857c-4374-8939-1eed2d487b4c-ca-trust-extracted\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.379642 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379569    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-slash\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.379821 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379645    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-bound-sa-token\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.379821 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379701    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.379821 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379784    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-multus\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.379956 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379841    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovnkube-config\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.379956 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379840    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2x9v\" (UniqueName: \"kubernetes.io/projected/53de1b04-dea0-4a6f-9609-760792cc0406-kube-api-access-f2x9v\") pod \"network-check-source-6f78d59f99-5s5g6\" (UID: \"53de1b04-dea0-4a6f-9609-760792cc0406\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6"
Oct 30 22:18:40.380056 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.379999    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.380104 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380071    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.380151 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380114    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-host\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.380202 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380173    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-trusted-ca\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.380249 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380214    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-installation-pull-secrets\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.380249 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380242    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvr4\" (UniqueName: \"kubernetes.io/projected/b40f9c33-c4fe-4176-9ae9-561592a770f6-kube-api-access-tgvr4\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.380460 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380439    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-cabundle\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:40.380560 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380485    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-trusted-ca\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:40.380638 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380585    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-cnibin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.380705 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380674    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-installation-pull-secrets\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.380742 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380722    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-snapshots\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.380793 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380775    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tndn\" (UniqueName: \"kubernetes.io/projected/1889bdc0-508e-4f97-80cb-4a5aa6e4f67f-kube-api-access-9tndn\") pod \"kas-connection-checker-685895c7c9-tsv8h\" (UID: \"1889bdc0-508e-4f97-80cb-4a5aa6e4f67f\") " pod="kube-system/kas-connection-checker-685895c7c9-tsv8h"
Oct 30 22:18:40.380915 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380897    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-stats-auth\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.380974 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.380931    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-run\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.381025 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381000    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-log-socket\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.381115 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381098    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-snapshots\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.381168 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381141    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9sn7\" (UniqueName: \"kubernetes.io/projected/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-api-access-t9sn7\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.381248 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381223    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.381248 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381230    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-trusted-ca\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.381359 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381282    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-stats-auth\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.381359 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381328    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-netns\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.381464 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381364    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5gx\" (UniqueName: \"kubernetes.io/projected/92a4974a-3e6e-4575-b5bd-303beb3fa201-kube-api-access-jp5gx\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:40.381640 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381584    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-trusted-ca\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:40.381721 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381705    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2vtz\" (UniqueName: \"kubernetes.io/projected/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-kube-api-access-p2vtz\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:40.381801 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381739    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/188a085b-a512-440b-90ec-30acc66e4171-cert\") pod \"ingress-canary-rvz8k\" (UID: \"188a085b-a512-440b-90ec-30acc66e4171\") " pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:40.381801 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381770    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-image-registry-private-configuration\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.381904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381803    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:40.381904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381827    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88c3c753-4382-452d-898d-732766a65bf1-serving-cert\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:40.381904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381855    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-hostroot\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.381904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381882    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-daemon-config\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.382116 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381911    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-trusted-ca-bundle\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:40.382116 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381934    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-socket-dir-parent\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.382116 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.381963    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-key\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:40.382274 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382225    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-image-registry-private-configuration\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.382274 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382240    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-trusted-ca-bundle\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:40.382374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382273    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a4974a-3e6e-4575-b5bd-303beb3fa201-tmp-dir\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:40.382374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382307    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-netd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.382374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382337    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-system-cni-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.382374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382366    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-kubelet\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.382607 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382433    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/188a085b-a512-440b-90ec-30acc66e4171-cert\") pod \"ingress-canary-rvz8k\" (UID: \"188a085b-a512-440b-90ec-30acc66e4171\") " pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:40.382607 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382437    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8199465-5b50-41fd-ab4f-5201977736a3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:40.382607 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382484    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-k8s-cni-cncf-io\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.382607 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382549    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-kube-api-access-rjxjz\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.382607 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382581    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq55l\" (UniqueName: \"kubernetes.io/projected/3a7d925c-af25-4c40-a280-e870b59cb772-kube-api-access-gq55l\") pod \"volume-data-source-validator-58c9f5f46d-tw8c8\" (UID: \"3a7d925c-af25-4c40-a280-e870b59cb772\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8"
Oct 30 22:18:40.382873 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382613    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.382873 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382629    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f8199465-5b50-41fd-ab4f-5201977736a3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:40.382873 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382643    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-tuned\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.382873 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382669    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.383100 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382936    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/afc45de8-4044-44c6-ac24-c39d8b2eece5-signing-key\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:40.383100 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.382967    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-daemon-config\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.383100 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383023    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-tuned\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.383100 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383092    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88c3c753-4382-452d-898d-732766a65bf1-serving-cert\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:40.383318 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383159    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92a4974a-3e6e-4575-b5bd-303beb3fa201-tmp-dir\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:40.383318 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383223    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88d59947-036f-412d-9ee6-f7565b0d4d8b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:40.383318 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383260    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvzt\" (UniqueName: \"kubernetes.io/projected/99252037-fc1d-4055-b66b-2f0e8ab910b8-kube-api-access-dlvzt\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.383463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383318    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-sys\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.383463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383381    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.383463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383439    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e71c29d-cd72-42ae-a0de-1500b78c89b0-tmp-dir\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:40.383640 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383469    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xd26\" (UniqueName: \"kubernetes.io/projected/8e71c29d-cd72-42ae-a0de-1500b78c89b0-kube-api-access-9xd26\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:40.383703 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383502    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-config\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:40.383703 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383659    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88d59947-036f-412d-9ee6-f7565b0d4d8b-nginx-conf\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:40.383794 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383777    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e71c29d-cd72-42ae-a0de-1500b78c89b0-tmp-dir\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:40.383894 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383878    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysconfig\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.383948 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383934    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-registry-certificates\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.383993 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.383968    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-serviceca\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:40.384087 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384067    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tdx8\" (UniqueName: \"kubernetes.io/projected/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-kube-api-access-7tdx8\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:40.384136 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384115    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-system-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.384177 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384151    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8199465-5b50-41fd-ab4f-5201977736a3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:40.384221 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384186    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f16a8be-4c89-4786-a385-ff738661c9bb-serving-cert\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:40.384262 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384221    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:40.384337 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384319    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v87rd\" (UniqueName: \"kubernetes.io/projected/83a06c56-145a-4a52-baca-2d01c679e581-kube-api-access-v87rd\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:40.384382 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384357    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:40.384426 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384389    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-systemd\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.384723 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384647    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgv2b\" (UniqueName: \"kubernetes.io/projected/012d43c7-e3f2-4931-877e-cdf66eaf61b6-kube-api-access-sgv2b\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.385112 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384887    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f8199465-5b50-41fd-ab4f-5201977736a3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:40.385112 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384968    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f16a8be-4c89-4786-a385-ff738661c9bb-serving-cert\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:40.385112 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384969    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-registry-certificates\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.385112 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.384989    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-serviceca\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:40.385320 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.385234    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88c3c753-4382-452d-898d-732766a65bf1-config\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:40.385552 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.385536    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovn-node-metrics-cert\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.385710 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.385212    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9af08ecc-3199-4c8b-b364-f42f4befeff1-ovn-node-metrics-cert\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.386021 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386008    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16ece3f9-76fd-4c93-b915-d1c8cec664fb-serving-cert\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.386052 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386032    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92gt\" (UniqueName: \"kubernetes.io/projected/88c3c753-4382-452d-898d-732766a65bf1-kube-api-access-j92gt\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:40.386106 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386051    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d45821a-5cff-41e5-b50c-90940f18d7c9-archives-path\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:40.386106 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386073    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-bound-sa-token\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.386106 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386097    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f16a8be-4c89-4786-a385-ff738661c9bb-config\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:40.386248 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386124    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92a4974a-3e6e-4575-b5bd-303beb3fa201-hosts-file\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:40.386248 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386161    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16ece3f9-76fd-4c93-b915-d1c8cec664fb-serving-cert\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.386248 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386174    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d45821a-5cff-41e5-b50c-90940f18d7c9-archives-path\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:40.386248 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386179    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-kubelet-config\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:40.386248 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386206    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-socket-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.386248 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386244    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-sys-fs\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.386501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386267    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.386501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386287    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-conf\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.386501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386316    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.386501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386333    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-bin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.386501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386366    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-certificates\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.386501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386390    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:40.386501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386408    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zptr\" (UniqueName: \"kubernetes.io/projected/de0170e7-99d6-4102-9b88-13552cb6f421-kube-api-access-2zptr\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.386501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386441    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f16a8be-4c89-4786-a385-ff738661c9bb-config\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:40.386501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386472    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-dbus\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386534    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-host\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386578    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386591    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-certificates\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386622    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-etc-kubernetes\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386640    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386701    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295mn\" (UniqueName: \"kubernetes.io/projected/afc45de8-4044-44c6-ac24-c39d8b2eece5-kube-api-access-295mn\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386731    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386767    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b40f9c33-c4fe-4176-9ae9-561592a770f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386774    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-default-certificate\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386830    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.386904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386875    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/407cf470-1b82-424c-a8e2-a5f061b3b84c-iptables-alerter-script\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386911    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-cni-binary-copy\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386944    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.386987    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-device-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387009    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-default-certificate\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387041    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387071    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-modprobe-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387086    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87c8cdc5-be2e-47a0-8cde-e944fda51056-cni-binary-copy\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387095    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60457f9c-65b0-4547-9012-2fc4efe2b9e8-konnectivity-ca\") pod \"konnectivity-agent-wb9bv\" (UID: \"60457f9c-65b0-4547-9012-2fc4efe2b9e8\") " pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387129    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzxg\" (UniqueName: \"kubernetes.io/projected/428997c4-b34e-4e3c-8551-c2d8510c69f4-kube-api-access-kqzxg\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387156    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/407cf470-1b82-424c-a8e2-a5f061b3b84c-iptables-alerter-script\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387219    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-installation-pull-secrets\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387239    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-tmp\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387270    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60457f9c-65b0-4547-9012-2fc4efe2b9e8-konnectivity-ca\") pod \"konnectivity-agent-wb9bv\" (UID: \"60457f9c-65b0-4547-9012-2fc4efe2b9e8\") " pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387313    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ece3f9-76fd-4c93-b915-d1c8cec664fb-tmp\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.387350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387331    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-original-pull-secret\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:40.387818 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.387416    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-installation-pull-secrets\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.388106 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.388086    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqfzc\" (UniqueName: \"kubernetes.io/projected/baf3e5ab-01d9-4caa-a239-b6df965e21e4-kube-api-access-nqfzc\") pod \"kas-connection-checker-685895c7c9-b4mzj\" (UID: \"baf3e5ab-01d9-4caa-a239-b6df965e21e4\") " pod="kube-system/kas-connection-checker-685895c7c9-b4mzj"
Oct 30 22:18:40.388158 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.388095    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8m5m\" (UniqueName: \"kubernetes.io/projected/2e346445-18a9-494c-9b3f-6c0860f631f7-kube-api-access-t8m5m\") pod \"kas-connection-checker-685895c7c9-8cbpq\" (UID: \"2e346445-18a9-494c-9b3f-6c0860f631f7\") " pod="kube-system/kas-connection-checker-685895c7c9-8cbpq"
Oct 30 22:18:40.388375 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.388361    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq7tw\" (UniqueName: \"kubernetes.io/projected/16ece3f9-76fd-4c93-b915-d1c8cec664fb-kube-api-access-nq7tw\") pod \"insights-operator-7c4986d449-xkcch\" (UID: \"16ece3f9-76fd-4c93-b915-d1c8cec664fb\") " pod="openshift-insights/insights-operator-7c4986d449-xkcch"
Oct 30 22:18:40.390418 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.390397    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vmt\" (UniqueName: \"kubernetes.io/projected/188a085b-a512-440b-90ec-30acc66e4171-kube-api-access-86vmt\") pod \"ingress-canary-rvz8k\" (UID: \"188a085b-a512-440b-90ec-30acc66e4171\") " pod="openshift-ingress-canary/ingress-canary-rvz8k"
Oct 30 22:18:40.392909 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.392890    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm75h\" (UniqueName: \"kubernetes.io/projected/87c8cdc5-be2e-47a0-8cde-e944fda51056-kube-api-access-wm75h\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.393001 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.392969    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kds\" (UniqueName: \"kubernetes.io/projected/f8199465-5b50-41fd-ab4f-5201977736a3-kube-api-access-v8kds\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b\" (UID: \"f8199465-5b50-41fd-ab4f-5201977736a3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b"
Oct 30 22:18:40.393785 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.393238    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpzp6\" (UniqueName: \"kubernetes.io/projected/16e7c643-41ee-4ac5-b285-1351ae323a1d-kube-api-access-mpzp6\") pod \"migrator-66df5d496b-ds7bd\" (UID: \"16e7c643-41ee-4ac5-b285-1351ae323a1d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ds7bd"
Oct 30 22:18:40.393785 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.393297    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvr4\" (UniqueName: \"kubernetes.io/projected/b40f9c33-c4fe-4176-9ae9-561592a770f6-kube-api-access-tgvr4\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.393785 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.393422    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62crk\" (UniqueName: \"kubernetes.io/projected/9f16a8be-4c89-4786-a385-ff738661c9bb-kube-api-access-62crk\") pod \"service-ca-operator-544d749c8c-2jl5h\" (UID: \"9f16a8be-4c89-4786-a385-ff738661c9bb\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h"
Oct 30 22:18:40.393785 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.393695    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tndn\" (UniqueName: \"kubernetes.io/projected/1889bdc0-508e-4f97-80cb-4a5aa6e4f67f-kube-api-access-9tndn\") pod \"kas-connection-checker-685895c7c9-tsv8h\" (UID: \"1889bdc0-508e-4f97-80cb-4a5aa6e4f67f\") " pod="kube-system/kas-connection-checker-685895c7c9-tsv8h"
Oct 30 22:18:40.393785 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.393753    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2x9v\" (UniqueName: \"kubernetes.io/projected/53de1b04-dea0-4a6f-9609-760792cc0406-kube-api-access-f2x9v\") pod \"network-check-source-6f78d59f99-5s5g6\" (UID: \"53de1b04-dea0-4a6f-9609-760792cc0406\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-5s5g6"
Oct 30 22:18:40.393973 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.393808    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9sn7\" (UniqueName: \"kubernetes.io/projected/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-api-access-t9sn7\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.393973 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.393864    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-bound-sa-token\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.393973 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.393943    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wnl2\" (UniqueName: \"kubernetes.io/projected/9d45821a-5cff-41e5-b50c-90940f18d7c9-kube-api-access-6wnl2\") pod \"periodic-gathering-qstj7-lntsk\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") " pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:18:40.397463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.397444    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq55l\" (UniqueName: \"kubernetes.io/projected/3a7d925c-af25-4c40-a280-e870b59cb772-kube-api-access-gq55l\") pod \"volume-data-source-validator-58c9f5f46d-tw8c8\" (UID: \"3a7d925c-af25-4c40-a280-e870b59cb772\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-tw8c8"
Oct 30 22:18:40.397535 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.397476    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7kc\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-kube-api-access-xl7kc\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.397654 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.397639    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrjv\" (UniqueName: \"kubernetes.io/projected/9af08ecc-3199-4c8b-b364-f42f4befeff1-kube-api-access-nkrjv\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.398286 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.398270    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdl48\" (UniqueName: \"kubernetes.io/projected/c097d835-0be1-4736-8dd8-98f375f02f47-kube-api-access-qdl48\") pod \"migrator-66df5d496b-vfc6j\" (UID: \"c097d835-0be1-4736-8dd8-98f375f02f47\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vfc6j"
Oct 30 22:18:40.398339 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.398325    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sfkt\" (UniqueName: \"kubernetes.io/projected/407cf470-1b82-424c-a8e2-a5f061b3b84c-kube-api-access-7sfkt\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:40.398992 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.398967    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2vtz\" (UniqueName: \"kubernetes.io/projected/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-kube-api-access-p2vtz\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:40.399054 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.398965    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5gx\" (UniqueName: \"kubernetes.io/projected/92a4974a-3e6e-4575-b5bd-303beb3fa201-kube-api-access-jp5gx\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:40.401864 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.401849    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-kube-api-access-rjxjz\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.401925 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.401891    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zptr\" (UniqueName: \"kubernetes.io/projected/de0170e7-99d6-4102-9b88-13552cb6f421-kube-api-access-2zptr\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.401982 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.401960    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-295mn\" (UniqueName: \"kubernetes.io/projected/afc45de8-4044-44c6-ac24-c39d8b2eece5-kube-api-access-295mn\") pod \"service-ca-5fffc97bdf-qlkzv\" (UID: \"afc45de8-4044-44c6-ac24-c39d8b2eece5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qlkzv"
Oct 30 22:18:40.402597 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.402582    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v87rd\" (UniqueName: \"kubernetes.io/projected/83a06c56-145a-4a52-baca-2d01c679e581-kube-api-access-v87rd\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:40.402926 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.402907    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvh42\" (UniqueName: \"kubernetes.io/projected/8eab1d96-88bb-417a-b03e-8e2f0274e862-kube-api-access-mvh42\") pod \"network-check-target-tgn5k\" (UID: \"8eab1d96-88bb-417a-b03e-8e2f0274e862\") " pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:40.403423 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.403404    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvzt\" (UniqueName: \"kubernetes.io/projected/99252037-fc1d-4055-b66b-2f0e8ab910b8-kube-api-access-dlvzt\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.403628 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.403610    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xd26\" (UniqueName: \"kubernetes.io/projected/8e71c29d-cd72-42ae-a0de-1500b78c89b0-kube-api-access-9xd26\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:40.404131 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.404114    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tdx8\" (UniqueName: \"kubernetes.io/projected/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-kube-api-access-7tdx8\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:40.404181 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.404155    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92gt\" (UniqueName: \"kubernetes.io/projected/88c3c753-4382-452d-898d-732766a65bf1-kube-api-access-j92gt\") pod \"console-operator-7cd56b8dbc-dd7n4\" (UID: \"88c3c753-4382-452d-898d-732766a65bf1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:40.404253 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.404238    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-bound-sa-token\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.406753 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.406733    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzxg\" (UniqueName: \"kubernetes.io/projected/428997c4-b34e-4e3c-8551-c2d8510c69f4-kube-api-access-kqzxg\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:40.406978 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.406964    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgv2b\" (UniqueName: \"kubernetes.io/projected/012d43c7-e3f2-4931-877e-cdf66eaf61b6-kube-api-access-sgv2b\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.487973 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.487941    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.488100 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.487982    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-lib-modules\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.488100 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488008    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-multus-certs\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488100 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488040    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-etc-selinux\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.488100 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488065    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-ovn\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.488100 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488096    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-var-lib-kubelet\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488117    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-systemd-units\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488139    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-slash\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488150    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-lib-modules\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488163    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-multus\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488206    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-ovn\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488218    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-var-lib-kubelet\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488214    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-slash\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488244    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-multus-certs\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488282    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488268    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-systemd-units\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488300    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-multus\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488329    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-host\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.488354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488345    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488368    5060 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488409    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-cnibin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488391    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-etc-selinux\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488358    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-cnibin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488422    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-host\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488468    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-run\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488492    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-log-socket\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488542    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488547    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-run\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488565    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-netns\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488586    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-log-socket\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488591    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488617    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-netns\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488629    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488633    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-hostroot\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488664    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-hostroot\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488665    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-socket-dir-parent\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488702    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-socket-dir-parent\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488700    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-netd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-netd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488774    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-system-cni-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488811    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-system-cni-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488813    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-kubelet\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488841    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-k8s-cni-cncf-io\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488863    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-kubelet\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488870    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.488899 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488914    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-run-k8s-cni-cncf-io\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488937    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488960    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-sys\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488982    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.488997    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489007    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysconfig\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489030    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-system-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489056    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489064    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-sys\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489078    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-systemd\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489105    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92a4974a-3e6e-4575-b5bd-303beb3fa201-hosts-file\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489127    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-kubelet-config\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489129    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysconfig\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489148    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-socket-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489167    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489170    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-sys-fs\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489205    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-systemd\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489219    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489211    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-sys-fs\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489244    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-conf\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92a4974a-3e6e-4575-b5bd-303beb3fa201-hosts-file\") pod \"node-resolver-qqwss\" (UID: \"92a4974a-3e6e-4575-b5bd-303beb3fa201\") " pod="openshift-dns/node-resolver-qqwss"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489270    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-system-cni-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489283    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489290    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-kubelet-config\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489308    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-bin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489333    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-dbus\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489347    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-socket-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.489345    5060 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489373    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-host\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489398    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-etc-kubernetes\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.489428    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls podName:83a06c56-145a-4a52-baca-2d01c679e581 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:40.989407813 +0000 UTC m=+1.809145625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r6fws" (UID: "83a06c56-145a-4a52-baca-2d01c679e581") : secret "samples-operator-tls" not found
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489447    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-etc-kubernetes\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489455    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489461    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489482    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489498    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-host-var-lib-cni-bin\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489528    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489542    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/adb5dbb9-f122-43fc-9ef9-867e6b2bb700-dbus\") pod \"global-pull-secret-syncer-xkt2l\" (UID: \"adb5dbb9-f122-43fc-9ef9-867e6b2bb700\") " pod="kube-system/global-pull-secret-syncer-xkt2l"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489550    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-device-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489573    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-modprobe-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489582    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6-host\") pod \"node-ca-7cjdl\" (UID: \"a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6\") " pod="openshift-image-registry/node-ca-7cjdl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489622    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-sysctl-conf\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489639    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-device-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.489709    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489743    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-modprobe-d\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489774    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-kubernetes\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.489789    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls podName:c69ae355-a045-48d1-b7c9-881d2c4e8a1b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:40.989774191 +0000 UTC m=+1.809511992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4vs7k" (UID: "c69ae355-a045-48d1-b7c9-881d2c4e8a1b") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489821    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/012d43c7-e3f2-4931-877e-cdf66eaf61b6-etc-kubernetes\") pod \"tuned-ggnd6\" (UID: \"012d43c7-e3f2-4931-877e-cdf66eaf61b6\") " pod="openshift-cluster-node-tuning-operator/tuned-ggnd6"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489814    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-var-lib-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489879    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489900    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-etc-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489925    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-cnibin\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489946    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/407cf470-1b82-424c-a8e2-a5f061b3b84c-host-slash\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489970    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-conf-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489994    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b40f9c33-c4fe-4176-9ae9-561592a770f6-cnibin\") pod \"multus-additional-cni-plugins-64cvm\" (UID: \"b40f9c33-c4fe-4176-9ae9-561592a770f6\") " pod="openshift-multus/multus-additional-cni-plugins-64cvm"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489995    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490026    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490030    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-kubelet\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490059    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3d84917-1245-4b8c-a887-fd2d668bcd2d-crio-socket\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490085    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490107    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490115    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/407cf470-1b82-424c-a8e2-a5f061b3b84c-host-slash\") pod \"iptables-alerter-vhkmj\" (UID: \"407cf470-1b82-424c-a8e2-a5f061b3b84c\") " pod="openshift-network-operator/iptables-alerter-vhkmj"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490131    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-node-log\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.489948    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-var-lib-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490168    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-etc-openvswitch\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490180    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87c8cdc5-be2e-47a0-8cde-e944fda51056-multus-conf-dir\") pod \"multus-kfshl\" (UID: \"87c8cdc5-be2e-47a0-8cde-e944fda51056\") " pod="openshift-multus/multus-kfshl"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490195    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-bin\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490221    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3d84917-1245-4b8c-a887-fd2d668bcd2d-crio-socket\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.490189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490233    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490262    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-registration-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490263    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3d84917-1245-4b8c-a887-fd2d668bcd2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490283    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-netns\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490318    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-systemd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490344    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490831    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99252037-fc1d-4055-b66b-2f0e8ab910b8-service-ca-bundle\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.490905    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.490943    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:40.990930004 +0000 UTC m=+1.810667815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490979    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-node-log\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.491017    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-cni-bin\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.490086    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-kubelet\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.491096    5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.491123    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls podName:8e71c29d-cd72-42ae-a0de-1500b78c89b0 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:40.991113333 +0000 UTC m=+1.810851160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls") pod "dns-default-6wrrt" (UID: "8e71c29d-cd72-42ae-a0de-1500b78c89b0") : secret "dns-default-metrics-tls" not found
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.491156    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-host-run-netns\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.491204    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") pod \"image-registry-5559797b78-pnvl9\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") " pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.491210    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de0170e7-99d6-4102-9b88-13552cb6f421-registration-dir\") pod \"aws-ebs-csi-driver-node-5z2tl\" (UID: \"de0170e7-99d6-4102-9b88-13552cb6f421\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z2tl"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.491257    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af08ecc-3199-4c8b-b364-f42f4befeff1-run-systemd\") pod \"ovnkube-node-7ppff\" (UID: \"9af08ecc-3199-4c8b-b364-f42f4befeff1\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.491662    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428997c4-b34e-4e3c-8551-c2d8510c69f4-metrics-certs\") pod \"network-metrics-daemon-hmgsj\" (UID: \"428997c4-b34e-4e3c-8551-c2d8510c69f4\") " pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.492819    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") pod \"image-registry-5bfb9545c9-tkcrm\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") " pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.492823    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88d59947-036f-412d-9ee6-f7565b0d4d8b-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grpzl\" (UID: \"88d59947-036f-412d-9ee6-f7565b0d4d8b\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:40.494887 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.494627    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99252037-fc1d-4055-b66b-2f0e8ab910b8-metrics-certs\") pod \"router-default-6668ff59b7-vglrn\" (UID: \"99252037-fc1d-4055-b66b-2f0e8ab910b8\") " pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.568821 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.568796    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:40.569363 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.569346    5060 scope.go:122] "RemoveContainer" containerID="6b06fc816b5d2acc17993f19f7a4cabafe3351bda1a804c24f06f570d748d8af"
Oct 30 22:18:40.569648 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.569610    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:40.570316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.570300    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl"
Oct 30 22:18:40.571465 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.571450    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hmgsj"
Oct 30 22:18:40.577704 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.577688    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:40.876933 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.876896    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6668ff59b7-vglrn"]
Oct 30 22:18:40.880371 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:40.880342    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99252037_fc1d_4055_b66b_2f0e8ab910b8.slice/crio-58b5622b4e2416455920e73de54d5538786e1140d18a24578d3abbb51cdd5b93 WatchSource:0}: Error finding container 58b5622b4e2416455920e73de54d5538786e1140d18a24578d3abbb51cdd5b93: Status 404 returned error can't find the container with id 58b5622b4e2416455920e73de54d5538786e1140d18a24578d3abbb51cdd5b93
Oct 30 22:18:40.914562 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.914531    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bfb9545c9-tkcrm"]
Oct 30 22:18:40.918262 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:40.918235    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4504af_560b_4e5a_a3cf_befff8d8d6d2.slice/crio-fec0ae7c30c778b5b7a2e3aff05cb2d690883b3ecf35d24bbdef03e55a93cb5f WatchSource:0}: Error finding container fec0ae7c30c778b5b7a2e3aff05cb2d690883b3ecf35d24bbdef03e55a93cb5f: Status 404 returned error can't find the container with id fec0ae7c30c778b5b7a2e3aff05cb2d690883b3ecf35d24bbdef03e55a93cb5f
Oct 30 22:18:40.966570 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.965810    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5559797b78-pnvl9"]
Oct 30 22:18:40.970929 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:40.970901    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555a97b6_857c_4374_8939_1eed2d487b4c.slice/crio-6f40dc4eb9c93c967fc2c3e64d0c66b7a1c80e3e9bf4d612135f07e6bcc734e1 WatchSource:0}: Error finding container 6f40dc4eb9c93c967fc2c3e64d0c66b7a1c80e3e9bf4d612135f07e6bcc734e1: Status 404 returned error can't find the container with id 6f40dc4eb9c93c967fc2c3e64d0c66b7a1c80e3e9bf4d612135f07e6bcc734e1
Oct 30 22:18:40.994081 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.994057    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:40.994169 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.994092    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:40.994169 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.994143    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:40.994241 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:40.994169    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:40.994277 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.994255    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:40.994307 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.994300    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls podName:c69ae355-a045-48d1-b7c9-881d2c4e8a1b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:41.994287964 +0000 UTC m=+2.814025764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4vs7k" (UID: "c69ae355-a045-48d1-b7c9-881d2c4e8a1b") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:40.994351 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.994340    5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:18:40.994383 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.994367    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls podName:8e71c29d-cd72-42ae-a0de-1500b78c89b0 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:41.994357856 +0000 UTC m=+2.814095658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls") pod "dns-default-6wrrt" (UID: "8e71c29d-cd72-42ae-a0de-1500b78c89b0") : secret "dns-default-metrics-tls" not found
Oct 30 22:18:40.994416 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.994402    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:40.994444 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.994420    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:41.994414429 +0000 UTC m=+2.814152226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:40.994476 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.994448    5060 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:18:40.994476 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:40.994466    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls podName:83a06c56-145a-4a52-baca-2d01c679e581 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:41.994461037 +0000 UTC m=+2.814198834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r6fws" (UID: "83a06c56-145a-4a52-baca-2d01c679e581") : secret "samples-operator-tls" not found
Oct 30 22:18:41.002527 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.002477    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-grpzl"]
Oct 30 22:18:41.016711 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:41.016682    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d59947_036f_412d_9ee6_f7565b0d4d8b.slice/crio-288fb363cb4cf7b13ca90da4c6597e16e8d5cdbdc237e951527b878c254fb515 WatchSource:0}: Error finding container 288fb363cb4cf7b13ca90da4c6597e16e8d5cdbdc237e951527b878c254fb515: Status 404 returned error can't find the container with id 288fb363cb4cf7b13ca90da4c6597e16e8d5cdbdc237e951527b878c254fb515
Oct 30 22:18:41.019169 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.019152    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:18:41.030701 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.030677    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hmgsj"]
Oct 30 22:18:41.033291 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:41.033273    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428997c4_b34e_4e3c_8551_c2d8510c69f4.slice/crio-d0a718c2de653901aec15e74f800d79b11eb6237ab478f9abb0de9f8608e42f6 WatchSource:0}: Error finding container d0a718c2de653901aec15e74f800d79b11eb6237ab478f9abb0de9f8608e42f6: Status 404 returned error can't find the container with id d0a718c2de653901aec15e74f800d79b11eb6237ab478f9abb0de9f8608e42f6
Oct 30 22:18:41.348301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.348222    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6668ff59b7-vglrn" event={"ID":"99252037-fc1d-4055-b66b-2f0e8ab910b8","Type":"ContainerStarted","Data":"0974c3818074a4712f44841ca9e8bdd95b235e78de99be77ceea760564c1b071"}
Oct 30 22:18:41.348301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.348287    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6668ff59b7-vglrn" event={"ID":"99252037-fc1d-4055-b66b-2f0e8ab910b8","Type":"ContainerStarted","Data":"58b5622b4e2416455920e73de54d5538786e1140d18a24578d3abbb51cdd5b93"}
Oct 30 22:18:41.349688 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.349657    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hmgsj" event={"ID":"428997c4-b34e-4e3c-8551-c2d8510c69f4","Type":"ContainerStarted","Data":"d0a718c2de653901aec15e74f800d79b11eb6237ab478f9abb0de9f8608e42f6"}
Oct 30 22:18:41.350864 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.350836    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl" event={"ID":"88d59947-036f-412d-9ee6-f7565b0d4d8b","Type":"ContainerStarted","Data":"288fb363cb4cf7b13ca90da4c6597e16e8d5cdbdc237e951527b878c254fb515"}
Oct 30 22:18:41.352463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.352431    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm" event={"ID":"2d4504af-560b-4e5a-a3cf-befff8d8d6d2","Type":"ContainerStarted","Data":"3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a"}
Oct 30 22:18:41.352572 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.352468    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm" event={"ID":"2d4504af-560b-4e5a-a3cf-befff8d8d6d2","Type":"ContainerStarted","Data":"fec0ae7c30c778b5b7a2e3aff05cb2d690883b3ecf35d24bbdef03e55a93cb5f"}
Oct 30 22:18:41.354110 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.354092    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:18:41.354547 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.354530    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/1.log"
Oct 30 22:18:41.354637 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.354560    5060 generic.go:356] "Generic (PLEG): container finished" podID="88c3c753-4382-452d-898d-732766a65bf1" containerID="8da5e92f8046ab557cf179b034d39807c59b8ce9a3cbd7c44a6f55057ff3eda4" exitCode=255
Oct 30 22:18:41.354694 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.354633    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" event={"ID":"88c3c753-4382-452d-898d-732766a65bf1","Type":"ContainerDied","Data":"8da5e92f8046ab557cf179b034d39807c59b8ce9a3cbd7c44a6f55057ff3eda4"}
Oct 30 22:18:41.354694 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.354683    5060 scope.go:122] "RemoveContainer" containerID="6b06fc816b5d2acc17993f19f7a4cabafe3351bda1a804c24f06f570d748d8af"
Oct 30 22:18:41.354969 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.354950    5060 scope.go:122] "RemoveContainer" containerID="8da5e92f8046ab557cf179b034d39807c59b8ce9a3cbd7c44a6f55057ff3eda4"
Oct 30 22:18:41.355204 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:41.355168    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dd7n4_openshift-console-operator(88c3c753-4382-452d-898d-732766a65bf1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" podUID="88c3c753-4382-452d-898d-732766a65bf1"
Oct 30 22:18:41.356320 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.356294    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5559797b78-pnvl9" event={"ID":"555a97b6-857c-4374-8939-1eed2d487b4c","Type":"ContainerStarted","Data":"47e7506b53872e1c73076d0aa1ee310028b22cc434955daef15df30de18b082b"}
Oct 30 22:18:41.356398 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.356340    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5559797b78-pnvl9" event={"ID":"555a97b6-857c-4374-8939-1eed2d487b4c","Type":"ContainerStarted","Data":"6f40dc4eb9c93c967fc2c3e64d0c66b7a1c80e3e9bf4d612135f07e6bcc734e1"}
Oct 30 22:18:41.569753 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.569720    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:41.572801 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:41.572778    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:42.004837 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.004803    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:42.004837 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.004840    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:42.005121 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.004895    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:42.005121 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.004923    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:42.005121 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:42.004967    5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:18:42.005121 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:42.005001    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:42.005121 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:42.005028    5060 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:18:42.005121 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:42.005049    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:44.00503026 +0000 UTC m=+4.824768071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:42.005121 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:42.005066    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls podName:8e71c29d-cd72-42ae-a0de-1500b78c89b0 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:44.005057793 +0000 UTC m=+4.824795596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls") pod "dns-default-6wrrt" (UID: "8e71c29d-cd72-42ae-a0de-1500b78c89b0") : secret "dns-default-metrics-tls" not found
Oct 30 22:18:42.005121 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:42.005032    5060 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:42.005121 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:42.005079    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls podName:83a06c56-145a-4a52-baca-2d01c679e581 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:44.005072638 +0000 UTC m=+4.824810439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r6fws" (UID: "83a06c56-145a-4a52-baca-2d01c679e581") : secret "samples-operator-tls" not found
Oct 30 22:18:42.005121 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:42.005112    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls podName:c69ae355-a045-48d1-b7c9-881d2c4e8a1b nodeName:}" failed. No retries permitted until 2026-10-30 22:18:44.005095412 +0000 UTC m=+4.824833227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4vs7k" (UID: "c69ae355-a045-48d1-b7c9-881d2c4e8a1b") : secret "cluster-monitoring-operator-tls" not found
Oct 30 22:18:42.297492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.297461    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:18:42.361945 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.361907    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hmgsj" event={"ID":"428997c4-b34e-4e3c-8551-c2d8510c69f4","Type":"ContainerStarted","Data":"b601950b4e73f1c1ec46360ea4f66dc80c696c7893356c98e42fa4341b11039f"}
Oct 30 22:18:42.362388 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.361959    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hmgsj" event={"ID":"428997c4-b34e-4e3c-8551-c2d8510c69f4","Type":"ContainerStarted","Data":"3ccee2386b3b7141500bac347bd6ace7cf477781f1b105ea9890eddd56efdc72"}
Oct 30 22:18:42.363478 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.363452    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-grpzl" event={"ID":"88d59947-036f-412d-9ee6-f7565b0d4d8b","Type":"ContainerStarted","Data":"07b04a1f61faa73817f574dae34c47051e188ddd5ea578cbbe4c72ac182ecb52"}
Oct 30 22:18:42.365030 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.365011    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:18:42.365167 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.365152    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:18:42.960856 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:42.960831    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:43.367875 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:43.367809    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:18:43.974033 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:43.973958    5060 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:43.974315 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:43.974301    5060 scope.go:122] "RemoveContainer" containerID="8da5e92f8046ab557cf179b034d39807c59b8ce9a3cbd7c44a6f55057ff3eda4"
Oct 30 22:18:43.974520 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:43.974487    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dd7n4_openshift-console-operator(88c3c753-4382-452d-898d-732766a65bf1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" podUID="88c3c753-4382-452d-898d-732766a65bf1"
Oct 30 22:18:44.024272 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:44.024241    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:44.024464 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:44.024292    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:44.024464 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:44.024321    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:44.024464 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:44.024351    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:44.024464 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:44.024377    5060 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 22:18:44.024464 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:44.024429    5060 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:18:44.024464 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:44.024460    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls podName:83a06c56-145a-4a52-baca-2d01c679e581 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:48.024438048 +0000 UTC m=+8.844175847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-r6fws" (UID: "83a06c56-145a-4a52-baca-2d01c679e581") : secret "samples-operator-tls" not found
Oct 30 22:18:44.024814 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:44.024496    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls podName:8e71c29d-cd72-42ae-a0de-1500b78c89b0 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:48.024479791 +0000 UTC m=+8.844217590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls") pod "dns-default-6wrrt" (UID: "8e71c29d-cd72-42ae-a0de-1500b78c89b0") : secret "dns-default-metrics-tls" not found
Oct 30 22:18:44.024814 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:44.024439    5060 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:44.024814 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:44.024573    5060 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls podName:d3d84917-1245-4b8c-a887-fd2d668bcd2d nodeName:}" failed. No retries permitted until 2026-10-30 22:18:48.024561738 +0000 UTC m=+8.844299536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6mwl7" (UID: "d3d84917-1245-4b8c-a887-fd2d668bcd2d") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:18:44.026717 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:44.026701    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c69ae355-a045-48d1-b7c9-881d2c4e8a1b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4vs7k\" (UID: \"c69ae355-a045-48d1-b7c9-881d2c4e8a1b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:44.168077 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:44.168044    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"
Oct 30 22:18:44.375418 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:44.375341    5060 generic.go:356] "Generic (PLEG): container finished" podID="f8199465-5b50-41fd-ab4f-5201977736a3" containerID="f2d55e4dd5911f41dd623deb5c81a34971d1eadc1c3c140e8e0c840be55f0de3" exitCode=0
Oct 30 22:18:44.375787 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:44.375418    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b" event={"ID":"f8199465-5b50-41fd-ab4f-5201977736a3","Type":"ContainerDied","Data":"f2d55e4dd5911f41dd623deb5c81a34971d1eadc1c3c140e8e0c840be55f0de3"}
Oct 30 22:18:44.375787 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:44.375769    5060 scope.go:122] "RemoveContainer" containerID="f2d55e4dd5911f41dd623deb5c81a34971d1eadc1c3c140e8e0c840be55f0de3"
Oct 30 22:18:44.440229 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:44.440202    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k"]
Oct 30 22:18:44.444002 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:44.443978    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69ae355_a045_48d1_b7c9_881d2c4e8a1b.slice/crio-cbfa68a5e5463e180c5e7fa70fdbfb1e7daee5bc484987c21251c1ae9106e218 WatchSource:0}: Error finding container cbfa68a5e5463e180c5e7fa70fdbfb1e7daee5bc484987c21251c1ae9106e218: Status 404 returned error can't find the container with id cbfa68a5e5463e180c5e7fa70fdbfb1e7daee5bc484987c21251c1ae9106e218
Oct 30 22:18:45.381066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:45.381024    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k" event={"ID":"c69ae355-a045-48d1-b7c9-881d2c4e8a1b","Type":"ContainerStarted","Data":"cbfa68a5e5463e180c5e7fa70fdbfb1e7daee5bc484987c21251c1ae9106e218"}
Oct 30 22:18:45.383193 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:45.383161    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b" event={"ID":"f8199465-5b50-41fd-ab4f-5201977736a3","Type":"ContainerStarted","Data":"90717b59fcbc76035114d54d8fd4528f6d4dc09b24e84d12d3b5b5eab6bb95aa"}
Oct 30 22:18:45.614679 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:45.614643    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:45.614865 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:45.614782    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:18:45.616172 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:45.616142    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6668ff59b7-vglrn"
Oct 30 22:18:46.388707 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:46.388631    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4vs7k" event={"ID":"c69ae355-a045-48d1-b7c9-881d2c4e8a1b","Type":"ContainerStarted","Data":"3a2ac1dbc8dbe14ee6b0fdc00c19dfafe00174542c4d623bf203f5564713d758"}
Oct 30 22:18:47.296217 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:47.296188    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:18:47.663343 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:47.663264    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:47.663728 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:47.663696    5060 scope.go:122] "RemoveContainer" containerID="8da5e92f8046ab557cf179b034d39807c59b8ce9a3cbd7c44a6f55057ff3eda4"
Oct 30 22:18:47.663888 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:18:47.663869    5060 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-dd7n4_openshift-console-operator(88c3c753-4382-452d-898d-732766a65bf1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" podUID="88c3c753-4382-452d-898d-732766a65bf1"
Oct 30 22:18:48.015293 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.015261    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:18:48.059371 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.059341    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:48.059547 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.059380    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:48.059547 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.059451    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:48.061819 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.061796    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e71c29d-cd72-42ae-a0de-1500b78c89b0-metrics-tls\") pod \"dns-default-6wrrt\" (UID: \"8e71c29d-cd72-42ae-a0de-1500b78c89b0\") " pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:48.061938 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.061804    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3d84917-1245-4b8c-a887-fd2d668bcd2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6mwl7\" (UID: \"d3d84917-1245-4b8c-a887-fd2d668bcd2d\") " pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:48.061938 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.061921    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/83a06c56-145a-4a52-baca-2d01c679e581-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-r6fws\" (UID: \"83a06c56-145a-4a52-baca-2d01c679e581\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:48.067817 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.067790    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"
Oct 30 22:18:48.085714 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.085691    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:48.086758 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.086738    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6mwl7"
Oct 30 22:18:48.352964 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.352895    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws"]
Oct 30 22:18:48.383803 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.383752    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6wrrt"]
Oct 30 22:18:48.386979 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:48.386946    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e71c29d_cd72_42ae_a0de_1500b78c89b0.slice/crio-ed2fcd84c0029071beb725050554cc186af4fd59d5599897293175fea289091d WatchSource:0}: Error finding container ed2fcd84c0029071beb725050554cc186af4fd59d5599897293175fea289091d: Status 404 returned error can't find the container with id ed2fcd84c0029071beb725050554cc186af4fd59d5599897293175fea289091d
Oct 30 22:18:48.387947 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.387923    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6mwl7"]
Oct 30 22:18:48.389696 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:48.389671    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d84917_1245_4b8c_a887_fd2d668bcd2d.slice/crio-ebe232dd901c1cc6b762cae45861e24d9ea12eabeb8e55d2e1461afd634d1d8c WatchSource:0}: Error finding container ebe232dd901c1cc6b762cae45861e24d9ea12eabeb8e55d2e1461afd634d1d8c: Status 404 returned error can't find the container with id ebe232dd901c1cc6b762cae45861e24d9ea12eabeb8e55d2e1461afd634d1d8c
Oct 30 22:18:48.395863 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.395840    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6wrrt" event={"ID":"8e71c29d-cd72-42ae-a0de-1500b78c89b0","Type":"ContainerStarted","Data":"ed2fcd84c0029071beb725050554cc186af4fd59d5599897293175fea289091d"}
Oct 30 22:18:48.397019 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:48.396998    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mwl7" event={"ID":"d3d84917-1245-4b8c-a887-fd2d668bcd2d","Type":"ContainerStarted","Data":"ebe232dd901c1cc6b762cae45861e24d9ea12eabeb8e55d2e1461afd634d1d8c"}
Oct 30 22:18:49.346687 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:49.346487    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:49.347089 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:49.346886    5060 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:18:49.402427 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:49.402393    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws" event={"ID":"83a06c56-145a-4a52-baca-2d01c679e581","Type":"ContainerStarted","Data":"43d2a1a1a215256ed63669ef6b954f647d85176e82ab5e3595484034ede1e8c4"}
Oct 30 22:18:49.404537 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:49.404489    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mwl7" event={"ID":"d3d84917-1245-4b8c-a887-fd2d668bcd2d","Type":"ContainerStarted","Data":"156c8e9c1ada39e42624841ecb6cb5a2e26dfb9965847c9b364de7f7680e6ec7"}
Oct 30 22:18:49.404658 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:49.404570    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mwl7" event={"ID":"d3d84917-1245-4b8c-a887-fd2d668bcd2d","Type":"ContainerStarted","Data":"6e51d2e8c836f613cd3a4d2d3f2d1dd67d02fe8de9503dc92283d2987710c6f1"}
Oct 30 22:18:51.414413 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:51.414376    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6wrrt" event={"ID":"8e71c29d-cd72-42ae-a0de-1500b78c89b0","Type":"ContainerStarted","Data":"bc7896d7a527e5582939441d3172dfd7b767e6de875291d8f4a1b1c8c0757903"}
Oct 30 22:18:51.414817 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:51.414442    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6wrrt" event={"ID":"8e71c29d-cd72-42ae-a0de-1500b78c89b0","Type":"ContainerStarted","Data":"21408d1cf99b7c229b6c521e24114c21600636e09ef0efdeaf046b9a551d2d4d"}
Oct 30 22:18:51.414817 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:51.414493    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:18:51.416229 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:51.416198    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws" event={"ID":"83a06c56-145a-4a52-baca-2d01c679e581","Type":"ContainerStarted","Data":"4a3931d6e4abb0ccdf80a01fd94daaf49956a57d8baa657de4790d954c33de56"}
Oct 30 22:18:51.416342 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:51.416242    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-r6fws" event={"ID":"83a06c56-145a-4a52-baca-2d01c679e581","Type":"ContainerStarted","Data":"912a0fc931cc28553cee79793897f6e9a970bdf89f51e8e1836fdb1f57ef2142"}
Oct 30 22:18:51.417995 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:51.417973    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6mwl7" event={"ID":"d3d84917-1245-4b8c-a887-fd2d668bcd2d","Type":"ContainerStarted","Data":"abb69b5f67b39c3c29335702b8f4e8f086776216c873f4201d78b3e5ecde1bbf"}
Oct 30 22:18:52.283751 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:52.283718    5060 scope.go:122] "RemoveContainer" containerID="8da5e92f8046ab557cf179b034d39807c59b8ce9a3cbd7c44a6f55057ff3eda4"
Oct 30 22:18:52.424152 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:52.424124    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:18:52.424632 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:52.424275    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4" event={"ID":"88c3c753-4382-452d-898d-732766a65bf1","Type":"ContainerStarted","Data":"71c8b649fd9cd2bf51e7050fdb3b26c1e5930abaf071f505f678dbdeea8e8a31"}
Oct 30 22:18:52.425105 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:52.425073    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:52.719399 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:52.719369    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-dd7n4"
Oct 30 22:18:54.133860 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.133820    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw"]
Oct 30 22:18:54.137964 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.137937    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw"
Oct 30 22:18:54.147037 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.147011    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 22:18:54.147218 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.147120    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6f4lz\""
Oct 30 22:18:54.154315 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.154292    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw"]
Oct 30 22:18:54.184423 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.184395    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bfb9545c9-tkcrm"]
Oct 30 22:18:54.214985 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.214953    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/18c9ba18-01c6-46e7-8e3c-de119ba81cc4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wwjmw\" (UID: \"18c9ba18-01c6-46e7-8e3c-de119ba81cc4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw"
Oct 30 22:18:54.236093 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.236065    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-7r6gx"]
Oct 30 22:18:54.239301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.239276    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7r6gx"
Oct 30 22:18:54.251601 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.251579    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 22:18:54.252331 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.252309    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 22:18:54.252432 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.252406    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7r6gx"]
Oct 30 22:18:54.252542 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.252499    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-xmh4p\""
Oct 30 22:18:54.315413 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.315378    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jskqs\" (UniqueName: \"kubernetes.io/projected/bd473000-e4aa-480c-bbf2-a15cfba05c40-kube-api-access-jskqs\") pod \"downloads-6dd789c8b7-7r6gx\" (UID: \"bd473000-e4aa-480c-bbf2-a15cfba05c40\") " pod="openshift-console/downloads-6dd789c8b7-7r6gx"
Oct 30 22:18:54.315604 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.315492    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd473000-e4aa-480c-bbf2-a15cfba05c40-tmp\") pod \"downloads-6dd789c8b7-7r6gx\" (UID: \"bd473000-e4aa-480c-bbf2-a15cfba05c40\") " pod="openshift-console/downloads-6dd789c8b7-7r6gx"
Oct 30 22:18:54.315604 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.315578    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/18c9ba18-01c6-46e7-8e3c-de119ba81cc4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wwjmw\" (UID: \"18c9ba18-01c6-46e7-8e3c-de119ba81cc4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw"
Oct 30 22:18:54.317988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.317966    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/18c9ba18-01c6-46e7-8e3c-de119ba81cc4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-wwjmw\" (UID: \"18c9ba18-01c6-46e7-8e3c-de119ba81cc4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw"
Oct 30 22:18:54.334415 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.334383    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c47bcc548-5vsbs"]
Oct 30 22:18:54.339599 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.339577    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.351328 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.351308    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c47bcc548-5vsbs"]
Oct 30 22:18:54.416155 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416058    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af0582ec-cfd0-41a7-be62-ad6b912e92f3-image-registry-private-configuration\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.416155 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416119    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af0582ec-cfd0-41a7-be62-ad6b912e92f3-ca-trust-extracted\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.416361 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416197    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd473000-e4aa-480c-bbf2-a15cfba05c40-tmp\") pod \"downloads-6dd789c8b7-7r6gx\" (UID: \"bd473000-e4aa-480c-bbf2-a15cfba05c40\") " pod="openshift-console/downloads-6dd789c8b7-7r6gx"
Oct 30 22:18:54.416361 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416234    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af0582ec-cfd0-41a7-be62-ad6b912e92f3-trusted-ca\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.416361 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416263    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af0582ec-cfd0-41a7-be62-ad6b912e92f3-registry-certificates\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.416361 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416285    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jskqs\" (UniqueName: \"kubernetes.io/projected/bd473000-e4aa-480c-bbf2-a15cfba05c40-kube-api-access-jskqs\") pod \"downloads-6dd789c8b7-7r6gx\" (UID: \"bd473000-e4aa-480c-bbf2-a15cfba05c40\") " pod="openshift-console/downloads-6dd789c8b7-7r6gx"
Oct 30 22:18:54.416361 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416303    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af0582ec-cfd0-41a7-be62-ad6b912e92f3-registry-tls\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.416558 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416469    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af0582ec-cfd0-41a7-be62-ad6b912e92f3-installation-pull-secrets\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.416558 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416542    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd473000-e4aa-480c-bbf2-a15cfba05c40-tmp\") pod \"downloads-6dd789c8b7-7r6gx\" (UID: \"bd473000-e4aa-480c-bbf2-a15cfba05c40\") " pod="openshift-console/downloads-6dd789c8b7-7r6gx"
Oct 30 22:18:54.416558 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416553    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjg6\" (UniqueName: \"kubernetes.io/projected/af0582ec-cfd0-41a7-be62-ad6b912e92f3-kube-api-access-wsjg6\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.416663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.416605    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af0582ec-cfd0-41a7-be62-ad6b912e92f3-bound-sa-token\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.436365 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.436333    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jskqs\" (UniqueName: \"kubernetes.io/projected/bd473000-e4aa-480c-bbf2-a15cfba05c40-kube-api-access-jskqs\") pod \"downloads-6dd789c8b7-7r6gx\" (UID: \"bd473000-e4aa-480c-bbf2-a15cfba05c40\") " pod="openshift-console/downloads-6dd789c8b7-7r6gx"
Oct 30 22:18:54.452155 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.452128    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw"
Oct 30 22:18:54.517782 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.517752    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af0582ec-cfd0-41a7-be62-ad6b912e92f3-ca-trust-extracted\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.517903 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.517798    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af0582ec-cfd0-41a7-be62-ad6b912e92f3-trusted-ca\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.517903 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.517822    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af0582ec-cfd0-41a7-be62-ad6b912e92f3-registry-certificates\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.517903 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.517847    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af0582ec-cfd0-41a7-be62-ad6b912e92f3-registry-tls\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.517903 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.517861    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af0582ec-cfd0-41a7-be62-ad6b912e92f3-installation-pull-secrets\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.518038 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.517903    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjg6\" (UniqueName: \"kubernetes.io/projected/af0582ec-cfd0-41a7-be62-ad6b912e92f3-kube-api-access-wsjg6\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.518038 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.517924    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af0582ec-cfd0-41a7-be62-ad6b912e92f3-bound-sa-token\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.518038 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.517955    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af0582ec-cfd0-41a7-be62-ad6b912e92f3-image-registry-private-configuration\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.518126 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.518112    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/af0582ec-cfd0-41a7-be62-ad6b912e92f3-ca-trust-extracted\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.519221 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.519192    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/af0582ec-cfd0-41a7-be62-ad6b912e92f3-trusted-ca\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.519536 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.519470    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/af0582ec-cfd0-41a7-be62-ad6b912e92f3-registry-certificates\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.521241 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.521222    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/af0582ec-cfd0-41a7-be62-ad6b912e92f3-installation-pull-secrets\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.521546 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.521526    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/af0582ec-cfd0-41a7-be62-ad6b912e92f3-registry-tls\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.521778 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.521758    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/af0582ec-cfd0-41a7-be62-ad6b912e92f3-image-registry-private-configuration\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.531190 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.531170    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af0582ec-cfd0-41a7-be62-ad6b912e92f3-bound-sa-token\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.531407 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.531389    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjg6\" (UniqueName: \"kubernetes.io/projected/af0582ec-cfd0-41a7-be62-ad6b912e92f3-kube-api-access-wsjg6\") pod \"image-registry-6c47bcc548-5vsbs\" (UID: \"af0582ec-cfd0-41a7-be62-ad6b912e92f3\") " pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.550405 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.550381    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7r6gx"
Oct 30 22:18:54.651786 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.651757    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:54.733833 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.733799    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw"]
Oct 30 22:18:54.736717 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:54.736692    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c9ba18_01c6_46e7_8e3c_de119ba81cc4.slice/crio-74bb155cc1751bf336dd4e94bab3f07872772a0534640a860951e833edd0073e WatchSource:0}: Error finding container 74bb155cc1751bf336dd4e94bab3f07872772a0534640a860951e833edd0073e: Status 404 returned error can't find the container with id 74bb155cc1751bf336dd4e94bab3f07872772a0534640a860951e833edd0073e
Oct 30 22:18:54.834156 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.834035    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7r6gx"]
Oct 30 22:18:54.836639 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:54.836609    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd473000_e4aa_480c_bbf2_a15cfba05c40.slice/crio-e6a348df78f5915e5f5b43e422bb439f8ecc3ad59f949d8799311077aa0e725a WatchSource:0}: Error finding container e6a348df78f5915e5f5b43e422bb439f8ecc3ad59f949d8799311077aa0e725a: Status 404 returned error can't find the container with id e6a348df78f5915e5f5b43e422bb439f8ecc3ad59f949d8799311077aa0e725a
Oct 30 22:18:54.934186 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:54.934159    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c47bcc548-5vsbs"]
Oct 30 22:18:54.936873 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:18:54.936847    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0582ec_cfd0_41a7_be62_ad6b912e92f3.slice/crio-c3ecddce7aed3b270c602bcbecdf6d14d1897c47b9dc8ac082af4b15d8e332e9 WatchSource:0}: Error finding container c3ecddce7aed3b270c602bcbecdf6d14d1897c47b9dc8ac082af4b15d8e332e9: Status 404 returned error can't find the container with id c3ecddce7aed3b270c602bcbecdf6d14d1897c47b9dc8ac082af4b15d8e332e9
Oct 30 22:18:55.084394 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:55.084368    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ppff"
Oct 30 22:18:55.439365 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:55.439328    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7r6gx" event={"ID":"bd473000-e4aa-480c-bbf2-a15cfba05c40","Type":"ContainerStarted","Data":"e6a348df78f5915e5f5b43e422bb439f8ecc3ad59f949d8799311077aa0e725a"}
Oct 30 22:18:55.441304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:55.441278    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs" event={"ID":"af0582ec-cfd0-41a7-be62-ad6b912e92f3","Type":"ContainerStarted","Data":"2b0c752dda1b485699cd7c5d93439a83534a510923593a56e6c18baf51b5e822"}
Oct 30 22:18:55.441434 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:55.441317    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs" event={"ID":"af0582ec-cfd0-41a7-be62-ad6b912e92f3","Type":"ContainerStarted","Data":"c3ecddce7aed3b270c602bcbecdf6d14d1897c47b9dc8ac082af4b15d8e332e9"}
Oct 30 22:18:55.441434 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:55.441363    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:18:55.442738 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:55.442705    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw" event={"ID":"18c9ba18-01c6-46e7-8e3c-de119ba81cc4","Type":"ContainerStarted","Data":"74bb155cc1751bf336dd4e94bab3f07872772a0534640a860951e833edd0073e"}
Oct 30 22:18:55.464880 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:55.464814    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs" podStartSLOduration=1.464800154 podStartE2EDuration="1.464800154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:55.464456011 +0000 UTC m=+16.284193832" watchObservedRunningTime="2026-10-30 22:18:55.464800154 +0000 UTC m=+16.284537983"
Oct 30 22:18:56.450141 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:56.450105    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw" event={"ID":"18c9ba18-01c6-46e7-8e3c-de119ba81cc4","Type":"ContainerStarted","Data":"6654c8fe0922bee2b3f13559fac3fce1d932f1491f4db7ca15667d98dc45b10b"}
Oct 30 22:18:56.450601 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:56.450340    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw"
Oct 30 22:18:56.456081 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:56.456048    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw"
Oct 30 22:18:56.472287 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:56.472233    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-wwjmw" podStartSLOduration=1.383389122 podStartE2EDuration="2.472214807s" totalImagesPullingTime="1.088825685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:56.470077283 +0000 UTC m=+17.289815108" watchObservedRunningTime="2026-10-30 22:18:56.472214807 +0000 UTC m=+17.291952628"
Oct 30 22:18:59.163423 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:59.163078    5060 patch_prober.go:29] interesting pod/image-registry-5559797b78-pnvl9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 22:18:59.163423 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:18:59.163151    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5559797b78-pnvl9" podUID="555a97b6-857c-4374-8939-1eed2d487b4c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:01.426888 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:01.426852    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6wrrt"
Oct 30 22:19:02.047581 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.047547    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5h8g5"]
Oct 30 22:19:02.053001 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.052962    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.104452 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.104355    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 22:19:02.104649 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.104489    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 22:19:02.104649 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.104565    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bh4rp\""
Oct 30 22:19:02.130034 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.130003    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 22:19:02.130203 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.130054    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 22:19:02.190560 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.190497    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-accelerators-collector-config\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.190729 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.190570    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-wtmp\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.190729 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.190627    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-tls\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.190729 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.190645    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-sys\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.190729 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.190664    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-root\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.190729 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.190688    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtksw\" (UniqueName: \"kubernetes.io/projected/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-kube-api-access-xtksw\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.190729 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.190708    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.191039 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.190742    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-textfile\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.191039 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.190764    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-metrics-client-ca\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292121 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292090    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-tls\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292125    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-sys\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292155    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-root\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292189    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtksw\" (UniqueName: \"kubernetes.io/projected/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-kube-api-access-xtksw\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292220    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292240    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-sys\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292248    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-root\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292260    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-textfile\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292304    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-metrics-client-ca\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292714 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292359    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-accelerators-collector-config\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292714 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292403    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-wtmp\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292714 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292595    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-textfile\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.292714 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.292617    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-wtmp\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.293200 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.293140    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-accelerators-collector-config\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.293200 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.293142    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-metrics-client-ca\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.295016 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.294994    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-tls\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.295016 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.295002    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.307862 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.307812    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtksw\" (UniqueName: \"kubernetes.io/projected/1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3-kube-api-access-xtksw\") pod \"node-exporter-5h8g5\" (UID: \"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3\") " pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.373578 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.373536    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5h8g5"
Oct 30 22:19:02.485457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:02.485420    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5h8g5" event={"ID":"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3","Type":"ContainerStarted","Data":"9e111fe559ce94d7bc41aeca53072230a243c51aa561ee36f7271b7c18ac4a98"}
Oct 30 22:19:03.374311 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:03.374279    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:19:04.190886 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:04.190860    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:19:04.497219 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:04.497182    5060 generic.go:356] "Generic (PLEG): container finished" podID="1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3" containerID="7dfc052da5c2aa5637760a47441537a86b006cadc4c90f8d4b97e8e26b05cc7d" exitCode=0
Oct 30 22:19:04.497390 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:04.497243    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5h8g5" event={"ID":"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3","Type":"ContainerDied","Data":"7dfc052da5c2aa5637760a47441537a86b006cadc4c90f8d4b97e8e26b05cc7d"}
Oct 30 22:19:04.778653 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:19:04.778528    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:19:05.504284 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:05.504237    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5h8g5" event={"ID":"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3","Type":"ContainerStarted","Data":"f04357ce95346612595f68b5b51a00841a6c0d32e6840017d3ba523598c334e6"}
Oct 30 22:19:05.504720 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:05.504311    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5h8g5" event={"ID":"1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3","Type":"ContainerStarted","Data":"7e02e11bc20573ffe276b2062058cc4e08ff6f87cc7b199d7be9e6e7bd3952d6"}
Oct 30 22:19:05.529921 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:05.529857    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5h8g5" podStartSLOduration=1.879368784 podStartE2EDuration="3.529839604s" totalImagesPullingTime="1.569377918s" totalInitContainerRuntime="81.092902ms" isStatefulPod=true podCreationTimestamp="2026-10-30 22:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:05.527762938 +0000 UTC m=+26.347500773" watchObservedRunningTime="2026-10-30 22:19:05.529839604 +0000 UTC m=+26.349577424"
Oct 30 22:19:08.019166 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:08.019132    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tgn5k"
Oct 30 22:19:10.312997 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:10.312967    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wb9bv"
Oct 30 22:19:11.536375 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:11.536342    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7r6gx" event={"ID":"bd473000-e4aa-480c-bbf2-a15cfba05c40","Type":"ContainerStarted","Data":"6676a63c2e711b976f15d1ce8fbe9dc199fb574098452f88ac6eab80d80b95f8"}
Oct 30 22:19:11.536759 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:11.536624    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-7r6gx"
Oct 30 22:19:11.538107 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:11.538083    5060 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-7r6gx container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body=
Oct 30 22:19:11.538220 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:11.538128    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-7r6gx" podUID="bd473000-e4aa-480c-bbf2-a15cfba05c40" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused"
Oct 30 22:19:11.558274 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:11.558234    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-7r6gx" podStartSLOduration=1.083933679 podStartE2EDuration="17.55822269s" totalImagesPullingTime="16.474289011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:11.557061558 +0000 UTC m=+32.376799368" watchObservedRunningTime="2026-10-30 22:19:11.55822269 +0000 UTC m=+32.377960507"
Oct 30 22:19:12.557006 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:12.556976    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-7r6gx"
Oct 30 22:19:16.454369 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:16.454334    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c47bcc548-5vsbs"
Oct 30 22:19:16.516925 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:16.516879    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5559797b78-pnvl9"]
Oct 30 22:19:19.206297 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.206254    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm" podUID="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" containerName="registry" containerID="cri-o://3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a" gracePeriod=30
Oct 30 22:19:19.515278 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.515186    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:19:19.558383 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.558332    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-ca-trust-extracted\") pod \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") "
Oct 30 22:19:19.558597 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.558390    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-installation-pull-secrets\") pod \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") "
Oct 30 22:19:19.558597 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.558425    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-certificates\") pod \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") "
Oct 30 22:19:19.558597 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.558487    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") pod \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") "
Oct 30 22:19:19.558597 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.558538    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-image-registry-private-configuration\") pod \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") "
Oct 30 22:19:19.558852 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.558607    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-kube-api-access-rjxjz\") pod \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") "
Oct 30 22:19:19.558852 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.558656    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-trusted-ca\") pod \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") "
Oct 30 22:19:19.558852 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.558716    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-bound-sa-token\") pod \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\" (UID: \"2d4504af-560b-4e5a-a3cf-befff8d8d6d2\") "
Oct 30 22:19:19.560655 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.560617    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-certificates" pod "2d4504af-560b-4e5a-a3cf-befff8d8d6d2" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:19.561458 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.561433    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-trusted-ca" pod "2d4504af-560b-4e5a-a3cf-befff8d8d6d2" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:19.568736 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.568706    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-ca-trust-extracted" pod "2d4504af-560b-4e5a-a3cf-befff8d8d6d2" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:19.571142 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.571091    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-kube-api-access-rjxjz" pod "2d4504af-560b-4e5a-a3cf-befff8d8d6d2" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2"). InnerVolumeSpecName "kube-api-access-rjxjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:19.572793 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.572168    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-installation-pull-secrets" pod "2d4504af-560b-4e5a-a3cf-befff8d8d6d2" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:19.574735 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.574690    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-image-registry-private-configuration" pod "2d4504af-560b-4e5a-a3cf-befff8d8d6d2" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:19.574838 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.574796    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-bound-sa-token" pod "2d4504af-560b-4e5a-a3cf-befff8d8d6d2" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:19.575151 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.575074    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls" pod "2d4504af-560b-4e5a-a3cf-befff8d8d6d2" (UID: "2d4504af-560b-4e5a-a3cf-befff8d8d6d2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:19.578999 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.578969    5060 generic.go:356] "Generic (PLEG): container finished" podID="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" containerID="3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a" exitCode=0
Oct 30 22:19:19.579112 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.579037    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm"
Oct 30 22:19:19.579176 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.579155    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm" event={"ID":"2d4504af-560b-4e5a-a3cf-befff8d8d6d2","Type":"ContainerDied","Data":"3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a"}
Oct 30 22:19:19.579260 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.579243    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfb9545c9-tkcrm" event={"ID":"2d4504af-560b-4e5a-a3cf-befff8d8d6d2","Type":"ContainerDied","Data":"fec0ae7c30c778b5b7a2e3aff05cb2d690883b3ecf35d24bbdef03e55a93cb5f"}
Oct 30 22:19:19.579342 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.579288    5060 scope.go:122] "RemoveContainer" containerID="3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a"
Oct 30 22:19:19.601648 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.601622    5060 scope.go:122] "RemoveContainer" containerID="3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a"
Oct 30 22:19:19.602022 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:19:19.601998    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a\": container with ID starting with 3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a not found: ID does not exist" containerID="3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a"
Oct 30 22:19:19.602117 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.602037    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a"} err="failed to get container status \"3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a\": rpc error: code = NotFound desc = could not find container \"3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a\": container with ID starting with 3f04fdb9a97d4cafedf12edfc29aeb8350cb4ca184fd5ee66f9380472198a69a not found: ID does not exist"
Oct 30 22:19:19.612844 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.612819    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bfb9545c9-tkcrm"]
Oct 30 22:19:19.619092 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.619059    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bfb9545c9-tkcrm"]
Oct 30 22:19:19.659669 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.659634    5060 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-ca-trust-extracted\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.659669 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.659664    5060 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-installation-pull-secrets\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.659871 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.659685    5060 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-certificates\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.659871 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.659700    5060 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-registry-tls\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.659871 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.659716    5060 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-image-registry-private-configuration\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.659871 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.659725    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjxjz\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-kube-api-access-rjxjz\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.659871 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.659734    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-trusted-ca\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.659871 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:19.659742    5060 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d4504af-560b-4e5a-a3cf-befff8d8d6d2-bound-sa-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:21.289144 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:21.289072    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" path="/var/lib/kubelet/pods/2d4504af-560b-4e5a-a3cf-befff8d8d6d2/volumes"
Oct 30 22:19:34.775637 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:19:34.775591    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:19:41.542711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.542649    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5559797b78-pnvl9" podUID="555a97b6-857c-4374-8939-1eed2d487b4c" containerName="registry" containerID="cri-o://47e7506b53872e1c73076d0aa1ee310028b22cc434955daef15df30de18b082b" gracePeriod=30
Oct 30 22:19:41.676291 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.676267    5060 generic.go:356] "Generic (PLEG): container finished" podID="555a97b6-857c-4374-8939-1eed2d487b4c" containerID="47e7506b53872e1c73076d0aa1ee310028b22cc434955daef15df30de18b082b" exitCode=0
Oct 30 22:19:41.676386 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.676338    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5559797b78-pnvl9" event={"ID":"555a97b6-857c-4374-8939-1eed2d487b4c","Type":"ContainerDied","Data":"47e7506b53872e1c73076d0aa1ee310028b22cc434955daef15df30de18b082b"}
Oct 30 22:19:41.830298 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.830270    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:19:41.954295 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.954257    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-trusted-ca\") pod \"555a97b6-857c-4374-8939-1eed2d487b4c\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") "
Oct 30 22:19:41.954469 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.954330    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-installation-pull-secrets\") pod \"555a97b6-857c-4374-8939-1eed2d487b4c\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") "
Oct 30 22:19:41.954469 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.954353    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-kube-api-access-xl7kc\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-kube-api-access-xl7kc\") pod \"555a97b6-857c-4374-8939-1eed2d487b4c\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") "
Oct 30 22:19:41.954469 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.954372    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-registry-certificates\") pod \"555a97b6-857c-4374-8939-1eed2d487b4c\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") "
Oct 30 22:19:41.954469 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.954392    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") pod \"555a97b6-857c-4374-8939-1eed2d487b4c\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") "
Oct 30 22:19:41.954706 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.954538    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555a97b6-857c-4374-8939-1eed2d487b4c-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/555a97b6-857c-4374-8939-1eed2d487b4c-ca-trust-extracted\") pod \"555a97b6-857c-4374-8939-1eed2d487b4c\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") "
Oct 30 22:19:41.954706 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.954591    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-bound-sa-token\") pod \"555a97b6-857c-4374-8939-1eed2d487b4c\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") "
Oct 30 22:19:41.954706 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.954624    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-image-registry-private-configuration\") pod \"555a97b6-857c-4374-8939-1eed2d487b4c\" (UID: \"555a97b6-857c-4374-8939-1eed2d487b4c\") "
Oct 30 22:19:41.954912 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.954866    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-trusted-ca" pod "555a97b6-857c-4374-8939-1eed2d487b4c" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:41.955070 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.954939    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-registry-certificates" pod "555a97b6-857c-4374-8939-1eed2d487b4c" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:41.956965 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.956918    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-installation-pull-secrets" pod "555a97b6-857c-4374-8939-1eed2d487b4c" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:41.957064 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.957010    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-bound-sa-token" pod "555a97b6-857c-4374-8939-1eed2d487b4c" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:41.957154 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.957130    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-kube-api-access-xl7kc" pod "555a97b6-857c-4374-8939-1eed2d487b4c" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c"). InnerVolumeSpecName "kube-api-access-xl7kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:41.957213 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.957138    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls" pod "555a97b6-857c-4374-8939-1eed2d487b4c" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:41.957270 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.957245    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-image-registry-private-configuration" pod "555a97b6-857c-4374-8939-1eed2d487b4c" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:41.961327 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:41.961302    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555a97b6-857c-4374-8939-1eed2d487b4c-ca-trust-extracted" pod "555a97b6-857c-4374-8939-1eed2d487b4c" (UID: "555a97b6-857c-4374-8939-1eed2d487b4c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:42.055745 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.055660    5060 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-installation-pull-secrets\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.055745 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.055692    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7kc\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-kube-api-access-xl7kc\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.055745 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.055702    5060 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-registry-certificates\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.055745 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.055713    5060 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-registry-tls\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.055745 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.055723    5060 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/555a97b6-857c-4374-8939-1eed2d487b4c-ca-trust-extracted\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.055745 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.055732    5060 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/555a97b6-857c-4374-8939-1eed2d487b4c-bound-sa-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.055745 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.055741    5060 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/555a97b6-857c-4374-8939-1eed2d487b4c-image-registry-private-configuration\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.055745 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.055752    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/555a97b6-857c-4374-8939-1eed2d487b4c-trusted-ca\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:42.682105 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.682074    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5559797b78-pnvl9" event={"ID":"555a97b6-857c-4374-8939-1eed2d487b4c","Type":"ContainerDied","Data":"6f40dc4eb9c93c967fc2c3e64d0c66b7a1c80e3e9bf4d612135f07e6bcc734e1"}
Oct 30 22:19:42.682105 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.682108    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5559797b78-pnvl9"
Oct 30 22:19:42.682709 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.682128    5060 scope.go:122] "RemoveContainer" containerID="47e7506b53872e1c73076d0aa1ee310028b22cc434955daef15df30de18b082b"
Oct 30 22:19:42.704208 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.704184    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5559797b78-pnvl9"]
Oct 30 22:19:42.708950 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:42.708930    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5559797b78-pnvl9"]
Oct 30 22:19:43.288960 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:43.288917    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555a97b6-857c-4374-8939-1eed2d487b4c" path="/var/lib/kubelet/pods/555a97b6-857c-4374-8939-1eed2d487b4c/volumes"
Oct 30 22:19:58.747080 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:58.747048    5060 generic.go:356] "Generic (PLEG): container finished" podID="16ece3f9-76fd-4c93-b915-d1c8cec664fb" containerID="25770bf587e8ffd6ba15cf576d9cf5a1e4962d9c7629f69ee3fd83ed388ff195" exitCode=0
Oct 30 22:19:58.747539 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:58.747094    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xkcch" event={"ID":"16ece3f9-76fd-4c93-b915-d1c8cec664fb","Type":"ContainerDied","Data":"25770bf587e8ffd6ba15cf576d9cf5a1e4962d9c7629f69ee3fd83ed388ff195"}
Oct 30 22:19:58.747539 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:58.747429    5060 scope.go:122] "RemoveContainer" containerID="25770bf587e8ffd6ba15cf576d9cf5a1e4962d9c7629f69ee3fd83ed388ff195"
Oct 30 22:19:59.752664 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:19:59.752625    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xkcch" event={"ID":"16ece3f9-76fd-4c93-b915-d1c8cec664fb","Type":"ContainerStarted","Data":"709ea1090392113832aa029f129af0e3f73b735351b79580523aa3ea423e8ddc"}
Oct 30 22:20:04.776047 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:20:04.776011    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:20:09.119523 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.119466    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-qstj7-lntsk"]
Oct 30 22:20:09.119915 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.119710    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-qstj7-lntsk" podUID="9d45821a-5cff-41e5-b50c-90940f18d7c9" containerName="insights-gathering" containerID="cri-o://c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442" gracePeriod=30
Oct 30 22:20:09.373165 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.373112    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:20:09.468034 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.467999    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d45821a-5cff-41e5-b50c-90940f18d7c9-kube-api-access-6wnl2\" (UniqueName: \"kubernetes.io/projected/9d45821a-5cff-41e5-b50c-90940f18d7c9-kube-api-access-6wnl2\") pod \"9d45821a-5cff-41e5-b50c-90940f18d7c9\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") "
Oct 30 22:20:09.468209 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.468043    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-trusted-ca-bundle\") pod \"9d45821a-5cff-41e5-b50c-90940f18d7c9\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") "
Oct 30 22:20:09.468209 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.468111    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d45821a-5cff-41e5-b50c-90940f18d7c9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d45821a-5cff-41e5-b50c-90940f18d7c9-archives-path\") pod \"9d45821a-5cff-41e5-b50c-90940f18d7c9\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") "
Oct 30 22:20:09.468209 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.468137    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-service-ca-bundle\") pod \"9d45821a-5cff-41e5-b50c-90940f18d7c9\" (UID: \"9d45821a-5cff-41e5-b50c-90940f18d7c9\") "
Oct 30 22:20:09.468540 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.468500    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-trusted-ca-bundle" pod "9d45821a-5cff-41e5-b50c-90940f18d7c9" (UID: "9d45821a-5cff-41e5-b50c-90940f18d7c9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:09.468602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.468498    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d45821a-5cff-41e5-b50c-90940f18d7c9-archives-path" pod "9d45821a-5cff-41e5-b50c-90940f18d7c9" (UID: "9d45821a-5cff-41e5-b50c-90940f18d7c9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:20:09.468602 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.468534    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-service-ca-bundle" pod "9d45821a-5cff-41e5-b50c-90940f18d7c9" (UID: "9d45821a-5cff-41e5-b50c-90940f18d7c9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:09.470297 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.470275    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d45821a-5cff-41e5-b50c-90940f18d7c9-kube-api-access-6wnl2" pod "9d45821a-5cff-41e5-b50c-90940f18d7c9" (UID: "9d45821a-5cff-41e5-b50c-90940f18d7c9"). InnerVolumeSpecName "kube-api-access-6wnl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:09.569296 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.569261    5060 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9d45821a-5cff-41e5-b50c-90940f18d7c9-archives-path\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:09.569296 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.569289    5060 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-service-ca-bundle\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:09.569296 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.569299    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wnl2\" (UniqueName: \"kubernetes.io/projected/9d45821a-5cff-41e5-b50c-90940f18d7c9-kube-api-access-6wnl2\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:09.569558 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.569309    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d45821a-5cff-41e5-b50c-90940f18d7c9-trusted-ca-bundle\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:09.792267 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.792232    5060 generic.go:356] "Generic (PLEG): container finished" podID="9d45821a-5cff-41e5-b50c-90940f18d7c9" containerID="c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442" exitCode=2
Oct 30 22:20:09.792464 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.792286    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qstj7-lntsk"
Oct 30 22:20:09.792464 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.792296    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qstj7-lntsk" event={"ID":"9d45821a-5cff-41e5-b50c-90940f18d7c9","Type":"ContainerDied","Data":"c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442"}
Oct 30 22:20:09.792464 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.792336    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qstj7-lntsk" event={"ID":"9d45821a-5cff-41e5-b50c-90940f18d7c9","Type":"ContainerDied","Data":"52e1662a1fb196c8242ec74e44da518dadad25943143c13f8984fdcd7795a6d4"}
Oct 30 22:20:09.792464 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.792354    5060 scope.go:122] "RemoveContainer" containerID="c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442"
Oct 30 22:20:09.800869 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.800851    5060 scope.go:122] "RemoveContainer" containerID="c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442"
Oct 30 22:20:09.801140 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:20:09.801121    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442\": container with ID starting with c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442 not found: ID does not exist" containerID="c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442"
Oct 30 22:20:09.801247 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.801148    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442"} err="failed to get container status \"c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442\": rpc error: code = NotFound desc = could not find container \"c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442\": container with ID starting with c8b322ad5bf74e6614baa8717f1bbc7261dda6723d0af728ec17a742a17e1442 not found: ID does not exist"
Oct 30 22:20:09.819288 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.819232    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-qstj7-lntsk"]
Oct 30 22:20:09.819288 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:09.819307    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-qstj7-lntsk"]
Oct 30 22:20:11.288696 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:11.288661    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d45821a-5cff-41e5-b50c-90940f18d7c9" path="/var/lib/kubelet/pods/9d45821a-5cff-41e5-b50c-90940f18d7c9/volumes"
Oct 30 22:20:14.814063 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:14.814030    5060 generic.go:356] "Generic (PLEG): container finished" podID="9f16a8be-4c89-4786-a385-ff738661c9bb" containerID="ff941d3d7ef8aefa21841303705314ad672e95364bd7943e557c850d759dba53" exitCode=0
Oct 30 22:20:14.814475 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:14.814109    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h" event={"ID":"9f16a8be-4c89-4786-a385-ff738661c9bb","Type":"ContainerDied","Data":"ff941d3d7ef8aefa21841303705314ad672e95364bd7943e557c850d759dba53"}
Oct 30 22:20:14.814475 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:14.814419    5060 scope.go:122] "RemoveContainer" containerID="ff941d3d7ef8aefa21841303705314ad672e95364bd7943e557c850d759dba53"
Oct 30 22:20:15.819730 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:20:15.819689    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2jl5h" event={"ID":"9f16a8be-4c89-4786-a385-ff738661c9bb","Type":"ContainerStarted","Data":"568c90f778d5c1207165236e65bb5283b4229d7446f1eb082b8366aa1aca8120"}
Oct 30 22:20:34.775467 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:20:34.775420    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:21:04.775775 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:21:04.775693    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:21:34.774789 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:21:34.774753    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:22:04.774219 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:22:04.774181    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:22:34.774870 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:22:34.774832    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:23:04.776865 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:23:04.776827    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:23:34.775529 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:23:34.775470    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:23:39.308013 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:23:39.307982    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:23:39.308499 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:23:39.308287    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:24:04.774892 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:24:04.774804    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:24:34.774703 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:24:34.774667    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:24:57.675785 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.675756    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"]
Oct 30 22:24:57.676266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.676048    5060 cpu_manager.go:414] "containerMap: removing container" podUID="555a97b6-857c-4374-8939-1eed2d487b4c" containerName="registry"
Oct 30 22:24:57.676266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.676058    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555a97b6-857c-4374-8939-1eed2d487b4c" containerName="registry"
Oct 30 22:24:57.676266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.676077    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9d45821a-5cff-41e5-b50c-90940f18d7c9" containerName="insights-gathering"
Oct 30 22:24:57.676266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.676083    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d45821a-5cff-41e5-b50c-90940f18d7c9" containerName="insights-gathering"
Oct 30 22:24:57.676266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.676094    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" containerName="registry"
Oct 30 22:24:57.676266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.676099    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" containerName="registry"
Oct 30 22:24:57.676266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.676148    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4504af-560b-4e5a-a3cf-befff8d8d6d2" containerName="registry"
Oct 30 22:24:57.676266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.676160    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="555a97b6-857c-4374-8939-1eed2d487b4c" containerName="registry"
Oct 30 22:24:57.676266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.676167    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d45821a-5cff-41e5-b50c-90940f18d7c9" containerName="insights-gathering"
Oct 30 22:24:57.677742 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.677728    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.726495 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.726465    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"]
Oct 30 22:24:57.778798 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.778764    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:24:57.778798 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.778770    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 22:24:57.779039 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.778842    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 22:24:57.780878 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.780862    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 22:24:57.780982 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.780893    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 22:24:57.781036 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.780986    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-t9fjn\""
Oct 30 22:24:57.835226 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.835199    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f872351-c46d-43a9-ba57-f7e80a2403f6-cert\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.835226 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.835226    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f872351-c46d-43a9-ba57-f7e80a2403f6-metrics-cert\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.835403 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.835310    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1f872351-c46d-43a9-ba57-f7e80a2403f6-manager-config\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.835403 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.835340    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s87g\" (UniqueName: \"kubernetes.io/projected/1f872351-c46d-43a9-ba57-f7e80a2403f6-kube-api-access-2s87g\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.936082 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.936012    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s87g\" (UniqueName: \"kubernetes.io/projected/1f872351-c46d-43a9-ba57-f7e80a2403f6-kube-api-access-2s87g\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.936082 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.936067    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f872351-c46d-43a9-ba57-f7e80a2403f6-cert\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.936239 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.936091    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f872351-c46d-43a9-ba57-f7e80a2403f6-metrics-cert\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.936239 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.936146    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1f872351-c46d-43a9-ba57-f7e80a2403f6-manager-config\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.936838 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.936816    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1f872351-c46d-43a9-ba57-f7e80a2403f6-manager-config\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.938561 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.938542    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f872351-c46d-43a9-ba57-f7e80a2403f6-metrics-cert\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.938631 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.938576    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f872351-c46d-43a9-ba57-f7e80a2403f6-cert\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.948873 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.948849    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s87g\" (UniqueName: \"kubernetes.io/projected/1f872351-c46d-43a9-ba57-f7e80a2403f6-kube-api-access-2s87g\") pod \"lws-controller-manager-6667976c75-tdwkq\" (UID: \"1f872351-c46d-43a9-ba57-f7e80a2403f6\") " pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:57.989613 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:57.989594    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:24:58.262683 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:58.262643    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"]
Oct 30 22:24:58.266476 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:24:58.266448    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f872351_c46d_43a9_ba57_f7e80a2403f6.slice/crio-874eed70045f837e6ac0137e3f7bb5e575245330a9578a48107a549e85b60b50 WatchSource:0}: Error finding container 874eed70045f837e6ac0137e3f7bb5e575245330a9578a48107a549e85b60b50: Status 404 returned error can't find the container with id 874eed70045f837e6ac0137e3f7bb5e575245330a9578a48107a549e85b60b50
Oct 30 22:24:58.268468 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:58.268452    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:24:58.908860 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:24:58.908823    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq" event={"ID":"1f872351-c46d-43a9-ba57-f7e80a2403f6","Type":"ContainerStarted","Data":"874eed70045f837e6ac0137e3f7bb5e575245330a9578a48107a549e85b60b50"}
Oct 30 22:25:02.929663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:02.929582    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq" event={"ID":"1f872351-c46d-43a9-ba57-f7e80a2403f6","Type":"ContainerStarted","Data":"82264323abbce21f93d171edc59f489784168e6a62c61d5bb283b9fc10a584c8"}
Oct 30 22:25:02.930092 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:02.929725    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:25:02.953767 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:02.953716    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq" podStartSLOduration=1.564933114 podStartE2EDuration="5.953703342s" totalImagesPullingTime="4.388770228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:02.952942134 +0000 UTC m=+383.772679954" watchObservedRunningTime="2026-10-30 22:25:02.953703342 +0000 UTC m=+383.773441154"
Oct 30 22:25:04.774264 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:25:04.774227    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:25:13.936574 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:13.936544    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6667976c75-tdwkq"
Oct 30 22:25:29.263916 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.263838    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t"]
Oct 30 22:25:29.270236 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.270217    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t"
Oct 30 22:25:29.276897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.276477    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpmtb\" (UniqueName: \"kubernetes.io/projected/43b7b42f-037e-43b2-96a9-424c67d04d69-kube-api-access-lpmtb\") pod \"dns-operator-controller-manager-68df894cb4-nk85t\" (UID: \"43b7b42f-037e-43b2-96a9-424c67d04d69\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t"
Oct 30 22:25:29.278315 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.278296    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bpsnm\""
Oct 30 22:25:29.280557 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.280538    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 22:25:29.280680 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.280665    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 22:25:29.280751 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.280737    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 22:25:29.282754 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.282734    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t"]
Oct 30 22:25:29.377383 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.377353    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpmtb\" (UniqueName: \"kubernetes.io/projected/43b7b42f-037e-43b2-96a9-424c67d04d69-kube-api-access-lpmtb\") pod \"dns-operator-controller-manager-68df894cb4-nk85t\" (UID: \"43b7b42f-037e-43b2-96a9-424c67d04d69\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t"
Oct 30 22:25:29.390659 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.390632    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpmtb\" (UniqueName: \"kubernetes.io/projected/43b7b42f-037e-43b2-96a9-424c67d04d69-kube-api-access-lpmtb\") pod \"dns-operator-controller-manager-68df894cb4-nk85t\" (UID: \"43b7b42f-037e-43b2-96a9-424c67d04d69\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t"
Oct 30 22:25:29.583760 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.583680    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t"
Oct 30 22:25:29.858006 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:29.857980    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t"]
Oct 30 22:25:29.860529 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:25:29.860488    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b7b42f_037e_43b2_96a9_424c67d04d69.slice/crio-4d9b3b353e90732fa143d73c4c9e071cc68eee950704ec61be297c727269f090 WatchSource:0}: Error finding container 4d9b3b353e90732fa143d73c4c9e071cc68eee950704ec61be297c727269f090: Status 404 returned error can't find the container with id 4d9b3b353e90732fa143d73c4c9e071cc68eee950704ec61be297c727269f090
Oct 30 22:25:30.036643 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:30.036606    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t" event={"ID":"43b7b42f-037e-43b2-96a9-424c67d04d69","Type":"ContainerStarted","Data":"4d9b3b353e90732fa143d73c4c9e071cc68eee950704ec61be297c727269f090"}
Oct 30 22:25:34.053577 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:34.053540    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t" event={"ID":"43b7b42f-037e-43b2-96a9-424c67d04d69","Type":"ContainerStarted","Data":"cf3cb412190496de10cc469001c44c84858eb9c62e78c68c05dcc2b6ae23a110"}
Oct 30 22:25:34.054057 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:34.053693    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t"
Oct 30 22:25:34.078796 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:34.078752    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t" podStartSLOduration=1.851710911 podStartE2EDuration="5.078740705s" totalImagesPullingTime="3.227029794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:34.077722332 +0000 UTC m=+414.897460154" watchObservedRunningTime="2026-10-30 22:25:34.078740705 +0000 UTC m=+414.898478523"
Oct 30 22:25:34.774913 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:25:34.774880    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:25:36.829623 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:36.829591    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"]
Oct 30 22:25:36.832834 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:36.832815    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"
Oct 30 22:25:36.837600 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:36.837581    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-972fs\""
Oct 30 22:25:36.853180 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:36.853158    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"]
Oct 30 22:25:36.938840 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:36.938805    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzcw\" (UniqueName: \"kubernetes.io/projected/bd14ba8d-e224-4d7d-827a-0f85607e435b-kube-api-access-9kzcw\") pod \"kuadrant-operator-controller-manager-658c459c99-2n266\" (UID: \"bd14ba8d-e224-4d7d-827a-0f85607e435b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"
Oct 30 22:25:36.939007 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:36.938849    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd14ba8d-e224-4d7d-827a-0f85607e435b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-658c459c99-2n266\" (UID: \"bd14ba8d-e224-4d7d-827a-0f85607e435b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"
Oct 30 22:25:37.039417 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:37.039382    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzcw\" (UniqueName: \"kubernetes.io/projected/bd14ba8d-e224-4d7d-827a-0f85607e435b-kube-api-access-9kzcw\") pod \"kuadrant-operator-controller-manager-658c459c99-2n266\" (UID: \"bd14ba8d-e224-4d7d-827a-0f85607e435b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"
Oct 30 22:25:37.039417 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:37.039422    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd14ba8d-e224-4d7d-827a-0f85607e435b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-658c459c99-2n266\" (UID: \"bd14ba8d-e224-4d7d-827a-0f85607e435b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"
Oct 30 22:25:37.039794 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:37.039777    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd14ba8d-e224-4d7d-827a-0f85607e435b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-658c459c99-2n266\" (UID: \"bd14ba8d-e224-4d7d-827a-0f85607e435b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"
Oct 30 22:25:37.053454 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:37.053422    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzcw\" (UniqueName: \"kubernetes.io/projected/bd14ba8d-e224-4d7d-827a-0f85607e435b-kube-api-access-9kzcw\") pod \"kuadrant-operator-controller-manager-658c459c99-2n266\" (UID: \"bd14ba8d-e224-4d7d-827a-0f85607e435b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"
Oct 30 22:25:37.144276 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:37.144207    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"
Oct 30 22:25:37.420266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:37.420132    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"]
Oct 30 22:25:37.423129 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:25:37.423102    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd14ba8d_e224_4d7d_827a_0f85607e435b.slice/crio-420eb0c63c20e272779f26f5cfb0a333f47d701c8fbee918b0d0c4eb5de7d0c8 WatchSource:0}: Error finding container 420eb0c63c20e272779f26f5cfb0a333f47d701c8fbee918b0d0c4eb5de7d0c8: Status 404 returned error can't find the container with id 420eb0c63c20e272779f26f5cfb0a333f47d701c8fbee918b0d0c4eb5de7d0c8
Oct 30 22:25:38.069521 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:38.069486    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266" event={"ID":"bd14ba8d-e224-4d7d-827a-0f85607e435b","Type":"ContainerStarted","Data":"420eb0c63c20e272779f26f5cfb0a333f47d701c8fbee918b0d0c4eb5de7d0c8"}
Oct 30 22:25:42.088953 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:42.088914    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266" event={"ID":"bd14ba8d-e224-4d7d-827a-0f85607e435b","Type":"ContainerStarted","Data":"3c96e765a85f1c605aeef72b35ea771888a5a4bbcfeec6f4bc356432a7364f76"}
Oct 30 22:25:42.089524 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:42.089051    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"
Oct 30 22:25:42.115948 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:42.115896    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266" podStartSLOduration=1.5667947770000001 podStartE2EDuration="6.115881396s" totalImagesPullingTime="4.549086619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:42.114498859 +0000 UTC m=+422.934236678" watchObservedRunningTime="2026-10-30 22:25:42.115881396 +0000 UTC m=+422.935619216"
Oct 30 22:25:44.421874 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.421839    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"]
Oct 30 22:25:44.424234 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.424214    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:44.436070 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.436051    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 22:25:44.436210 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.436079    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 22:25:44.436275 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.436118    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-b87l4\""
Oct 30 22:25:44.451127 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.451098    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"]
Oct 30 22:25:44.505115 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.505084    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e7668c50-a921-4e34-8ea3-9b84bb52b435-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pk9xx\" (UID: \"e7668c50-a921-4e34-8ea3-9b84bb52b435\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:44.505115 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.505114    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7668c50-a921-4e34-8ea3-9b84bb52b435-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pk9xx\" (UID: \"e7668c50-a921-4e34-8ea3-9b84bb52b435\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:44.505283 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.505136    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bq54\" (UniqueName: \"kubernetes.io/projected/e7668c50-a921-4e34-8ea3-9b84bb52b435-kube-api-access-6bq54\") pod \"kuadrant-console-plugin-66c4fb575d-pk9xx\" (UID: \"e7668c50-a921-4e34-8ea3-9b84bb52b435\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:44.605646 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.605611    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e7668c50-a921-4e34-8ea3-9b84bb52b435-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pk9xx\" (UID: \"e7668c50-a921-4e34-8ea3-9b84bb52b435\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:44.605779 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.605647    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7668c50-a921-4e34-8ea3-9b84bb52b435-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pk9xx\" (UID: \"e7668c50-a921-4e34-8ea3-9b84bb52b435\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:44.605779 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.605684    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bq54\" (UniqueName: \"kubernetes.io/projected/e7668c50-a921-4e34-8ea3-9b84bb52b435-kube-api-access-6bq54\") pod \"kuadrant-console-plugin-66c4fb575d-pk9xx\" (UID: \"e7668c50-a921-4e34-8ea3-9b84bb52b435\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:44.606377 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.606357    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e7668c50-a921-4e34-8ea3-9b84bb52b435-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pk9xx\" (UID: \"e7668c50-a921-4e34-8ea3-9b84bb52b435\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:44.608002 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.607982    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7668c50-a921-4e34-8ea3-9b84bb52b435-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pk9xx\" (UID: \"e7668c50-a921-4e34-8ea3-9b84bb52b435\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:44.619598 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.619569    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bq54\" (UniqueName: \"kubernetes.io/projected/e7668c50-a921-4e34-8ea3-9b84bb52b435-kube-api-access-6bq54\") pod \"kuadrant-console-plugin-66c4fb575d-pk9xx\" (UID: \"e7668c50-a921-4e34-8ea3-9b84bb52b435\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:44.750066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:44.750032    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"
Oct 30 22:25:45.022883 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:45.022806    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx"]
Oct 30 22:25:45.025655 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:25:45.025627    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7668c50_a921_4e34_8ea3_9b84bb52b435.slice/crio-51aea843b7a96edc344052b98f415ff2a7a4d14bdb644d6584aa7b45cf48443f WatchSource:0}: Error finding container 51aea843b7a96edc344052b98f415ff2a7a4d14bdb644d6584aa7b45cf48443f: Status 404 returned error can't find the container with id 51aea843b7a96edc344052b98f415ff2a7a4d14bdb644d6584aa7b45cf48443f
Oct 30 22:25:45.060268 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:45.060244    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-nk85t"
Oct 30 22:25:45.102703 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:45.102673    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx" event={"ID":"e7668c50-a921-4e34-8ea3-9b84bb52b435","Type":"ContainerStarted","Data":"51aea843b7a96edc344052b98f415ff2a7a4d14bdb644d6584aa7b45cf48443f"}
Oct 30 22:25:50.125923 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:50.125885    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx" event={"ID":"e7668c50-a921-4e34-8ea3-9b84bb52b435","Type":"ContainerStarted","Data":"d567a8de5ddbab0e5f7ccdba3bfe68535c6eab947000e140cb1bd02dc7283e9b"}
Oct 30 22:25:50.144913 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:50.144865    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pk9xx" podStartSLOduration=1.20444852 podStartE2EDuration="6.144853134s" totalImagesPullingTime="4.940404614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:50.144359598 +0000 UTC m=+430.964097419" watchObservedRunningTime="2026-10-30 22:25:50.144853134 +0000 UTC m=+430.964590968"
Oct 30 22:25:53.095014 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:25:53.094987    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-658c459c99-2n266"
Oct 30 22:26:04.777171 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:26:04.777136    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:26:34.775561 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:26:34.775522    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:26:41.023651 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.023601    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" podUID="a5c727ac326546386befce51c4d99c6d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 22:26:41.106565 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.106536    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c4bgj"]
Oct 30 22:26:41.129701 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.129667    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c4bgj"]
Oct 30 22:26:41.129852 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.129760    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c4bgj"
Oct 30 22:26:41.137143 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.137112    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rr875\""
Oct 30 22:26:41.167457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.167425    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5pr\" (UniqueName: \"kubernetes.io/projected/2d3965d6-4b34-4083-9b03-9937e901c20b-kube-api-access-rx5pr\") pod \"authorino-57f7f8659f-c4bgj\" (UID: \"2d3965d6-4b34-4083-9b03-9937e901c20b\") " pod="kuadrant-system/authorino-57f7f8659f-c4bgj"
Oct 30 22:26:41.255212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.255160    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gkt2p"]
Oct 30 22:26:41.258990 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.258974    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p"
Oct 30 22:26:41.268425 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.268383    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5pr\" (UniqueName: \"kubernetes.io/projected/2d3965d6-4b34-4083-9b03-9937e901c20b-kube-api-access-rx5pr\") pod \"authorino-57f7f8659f-c4bgj\" (UID: \"2d3965d6-4b34-4083-9b03-9937e901c20b\") " pod="kuadrant-system/authorino-57f7f8659f-c4bgj"
Oct 30 22:26:41.271412 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.271393    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gkt2p"]
Oct 30 22:26:41.285650 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.285577    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5pr\" (UniqueName: \"kubernetes.io/projected/2d3965d6-4b34-4083-9b03-9937e901c20b-kube-api-access-rx5pr\") pod \"authorino-57f7f8659f-c4bgj\" (UID: \"2d3965d6-4b34-4083-9b03-9937e901c20b\") " pod="kuadrant-system/authorino-57f7f8659f-c4bgj"
Oct 30 22:26:41.369245 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.369209    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknwc\" (UniqueName: \"kubernetes.io/projected/86239374-2333-40f4-b0cb-3e83d7b4cf4c-kube-api-access-wknwc\") pod \"authorino-56c6c9ddd5-gkt2p\" (UID: \"86239374-2333-40f4-b0cb-3e83d7b4cf4c\") " pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p"
Oct 30 22:26:41.447001 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.446966    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c4bgj"
Oct 30 22:26:41.470539 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.470492    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknwc\" (UniqueName: \"kubernetes.io/projected/86239374-2333-40f4-b0cb-3e83d7b4cf4c-kube-api-access-wknwc\") pod \"authorino-56c6c9ddd5-gkt2p\" (UID: \"86239374-2333-40f4-b0cb-3e83d7b4cf4c\") " pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p"
Oct 30 22:26:41.484291 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.484234    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknwc\" (UniqueName: \"kubernetes.io/projected/86239374-2333-40f4-b0cb-3e83d7b4cf4c-kube-api-access-wknwc\") pod \"authorino-56c6c9ddd5-gkt2p\" (UID: \"86239374-2333-40f4-b0cb-3e83d7b4cf4c\") " pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p"
Oct 30 22:26:41.570222 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.570140    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p"
Oct 30 22:26:41.717317 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.717136    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c4bgj"]
Oct 30 22:26:41.719265 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:26:41.719238    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d3965d6_4b34_4083_9b03_9937e901c20b.slice/crio-640a5889e3247b3b8cb79fe6d3000f49d89114bdc33835052b1dfd20e74574bd WatchSource:0}: Error finding container 640a5889e3247b3b8cb79fe6d3000f49d89114bdc33835052b1dfd20e74574bd: Status 404 returned error can't find the container with id 640a5889e3247b3b8cb79fe6d3000f49d89114bdc33835052b1dfd20e74574bd
Oct 30 22:26:41.842171 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:41.842102    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gkt2p"]
Oct 30 22:26:41.845013 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:26:41.844988    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86239374_2333_40f4_b0cb_3e83d7b4cf4c.slice/crio-b408fd88cddd00fd76d6298d6ad74221f9e4afd39166a94cd2d208c13c14ed2b WatchSource:0}: Error finding container b408fd88cddd00fd76d6298d6ad74221f9e4afd39166a94cd2d208c13c14ed2b: Status 404 returned error can't find the container with id b408fd88cddd00fd76d6298d6ad74221f9e4afd39166a94cd2d208c13c14ed2b
Oct 30 22:26:42.344051 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:42.344015    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c4bgj" event={"ID":"2d3965d6-4b34-4083-9b03-9937e901c20b","Type":"ContainerStarted","Data":"640a5889e3247b3b8cb79fe6d3000f49d89114bdc33835052b1dfd20e74574bd"}
Oct 30 22:26:42.345229 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:42.345204    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p" event={"ID":"86239374-2333-40f4-b0cb-3e83d7b4cf4c","Type":"ContainerStarted","Data":"b408fd88cddd00fd76d6298d6ad74221f9e4afd39166a94cd2d208c13c14ed2b"}
Oct 30 22:26:44.355864 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:44.355827    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p" event={"ID":"86239374-2333-40f4-b0cb-3e83d7b4cf4c","Type":"ContainerStarted","Data":"e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5"}
Oct 30 22:26:44.357215 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:44.357189    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c4bgj" event={"ID":"2d3965d6-4b34-4083-9b03-9937e901c20b","Type":"ContainerStarted","Data":"2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4"}
Oct 30 22:26:44.374626 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:44.374583    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p" podStartSLOduration=1.072208569 podStartE2EDuration="3.374569833s" totalImagesPullingTime="2.302361264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:44.373869577 +0000 UTC m=+485.193607397" watchObservedRunningTime="2026-10-30 22:26:44.374569833 +0000 UTC m=+485.194307652"
Oct 30 22:26:44.395722 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:44.395677    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-c4bgj" podStartSLOduration=1.200501651 podStartE2EDuration="3.395663525s" totalImagesPullingTime="2.195161874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:44.394163715 +0000 UTC m=+485.213901540" watchObservedRunningTime="2026-10-30 22:26:44.395663525 +0000 UTC m=+485.215401344"
Oct 30 22:26:44.402842 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:44.402808    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c4bgj"]
Oct 30 22:26:46.364647 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:46.364605    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-c4bgj" podUID="2d3965d6-4b34-4083-9b03-9937e901c20b" containerName="authorino" containerID="cri-o://2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4" gracePeriod=30
Oct 30 22:26:46.605547 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:46.605517    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c4bgj"
Oct 30 22:26:46.718060 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:46.718026    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3965d6-4b34-4083-9b03-9937e901c20b-kube-api-access-rx5pr\" (UniqueName: \"kubernetes.io/projected/2d3965d6-4b34-4083-9b03-9937e901c20b-kube-api-access-rx5pr\") pod \"2d3965d6-4b34-4083-9b03-9937e901c20b\" (UID: \"2d3965d6-4b34-4083-9b03-9937e901c20b\") "
Oct 30 22:26:46.720281 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:46.720258    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3965d6-4b34-4083-9b03-9937e901c20b-kube-api-access-rx5pr" pod "2d3965d6-4b34-4083-9b03-9937e901c20b" (UID: "2d3965d6-4b34-4083-9b03-9937e901c20b"). InnerVolumeSpecName "kube-api-access-rx5pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:46.819171 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:46.819127    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx5pr\" (UniqueName: \"kubernetes.io/projected/2d3965d6-4b34-4083-9b03-9937e901c20b-kube-api-access-rx5pr\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.369949 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:47.369913    5060 generic.go:356] "Generic (PLEG): container finished" podID="2d3965d6-4b34-4083-9b03-9937e901c20b" containerID="2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4" exitCode=0
Oct 30 22:26:47.370347 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:47.369956    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c4bgj"
Oct 30 22:26:47.370347 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:47.369969    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c4bgj" event={"ID":"2d3965d6-4b34-4083-9b03-9937e901c20b","Type":"ContainerDied","Data":"2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4"}
Oct 30 22:26:47.370347 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:47.370006    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c4bgj" event={"ID":"2d3965d6-4b34-4083-9b03-9937e901c20b","Type":"ContainerDied","Data":"640a5889e3247b3b8cb79fe6d3000f49d89114bdc33835052b1dfd20e74574bd"}
Oct 30 22:26:47.370347 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:47.370025    5060 scope.go:122] "RemoveContainer" containerID="2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4"
Oct 30 22:26:47.377966 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:47.377949    5060 scope.go:122] "RemoveContainer" containerID="2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4"
Oct 30 22:26:47.378194 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:26:47.378177    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4\": container with ID starting with 2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4 not found: ID does not exist" containerID="2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4"
Oct 30 22:26:47.378286 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:47.378202    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4"} err="failed to get container status \"2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4\": rpc error: code = NotFound desc = could not find container \"2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4\": container with ID starting with 2b5455f0929acc4d4e82c58279db931add33874fecb41edd062569a4cb9b12a4 not found: ID does not exist"
Oct 30 22:26:47.389338 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:47.389318    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c4bgj"]
Oct 30 22:26:47.393684 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:47.393661    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c4bgj"]
Oct 30 22:26:49.289277 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:26:49.289245    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3965d6-4b34-4083-9b03-9937e901c20b" path="/var/lib/kubelet/pods/2d3965d6-4b34-4083-9b03-9937e901c20b/volumes"
Oct 30 22:27:04.774863 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:27:04.774789    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:27:08.902574 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:08.902540    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gkt2p"]
Oct 30 22:27:08.902955 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:08.902775    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p" podUID="86239374-2333-40f4-b0cb-3e83d7b4cf4c" containerName="authorino" containerID="cri-o://e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5" gracePeriod=30
Oct 30 22:27:09.144707 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.144683    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p"
Oct 30 22:27:09.319161 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.319126    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86239374-2333-40f4-b0cb-3e83d7b4cf4c-kube-api-access-wknwc\" (UniqueName: \"kubernetes.io/projected/86239374-2333-40f4-b0cb-3e83d7b4cf4c-kube-api-access-wknwc\") pod \"86239374-2333-40f4-b0cb-3e83d7b4cf4c\" (UID: \"86239374-2333-40f4-b0cb-3e83d7b4cf4c\") "
Oct 30 22:27:09.321131 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.321111    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86239374-2333-40f4-b0cb-3e83d7b4cf4c-kube-api-access-wknwc" pod "86239374-2333-40f4-b0cb-3e83d7b4cf4c" (UID: "86239374-2333-40f4-b0cb-3e83d7b4cf4c"). InnerVolumeSpecName "kube-api-access-wknwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:09.420021 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.419987    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wknwc\" (UniqueName: \"kubernetes.io/projected/86239374-2333-40f4-b0cb-3e83d7b4cf4c-kube-api-access-wknwc\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:09.464529 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.464479    5060 generic.go:356] "Generic (PLEG): container finished" podID="86239374-2333-40f4-b0cb-3e83d7b4cf4c" containerID="e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5" exitCode=0
Oct 30 22:27:09.464688 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.464550    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p"
Oct 30 22:27:09.464688 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.464567    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p" event={"ID":"86239374-2333-40f4-b0cb-3e83d7b4cf4c","Type":"ContainerDied","Data":"e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5"}
Oct 30 22:27:09.464688 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.464612    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gkt2p" event={"ID":"86239374-2333-40f4-b0cb-3e83d7b4cf4c","Type":"ContainerDied","Data":"b408fd88cddd00fd76d6298d6ad74221f9e4afd39166a94cd2d208c13c14ed2b"}
Oct 30 22:27:09.464688 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.464635    5060 scope.go:122] "RemoveContainer" containerID="e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5"
Oct 30 22:27:09.473232 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.473217    5060 scope.go:122] "RemoveContainer" containerID="e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5"
Oct 30 22:27:09.473459 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:27:09.473443    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5\": container with ID starting with e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5 not found: ID does not exist" containerID="e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5"
Oct 30 22:27:09.473619 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.473477    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5"} err="failed to get container status \"e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5\": rpc error: code = NotFound desc = could not find container \"e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5\": container with ID starting with e9652823fe8c23fd6364edd6a428473d40639e1e42eab3ec9cf2c214cbc43ff5 not found: ID does not exist"
Oct 30 22:27:09.486921 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.486903    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gkt2p"]
Oct 30 22:27:09.493556 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:09.493539    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gkt2p"]
Oct 30 22:27:11.289295 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:11.289263    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86239374-2333-40f4-b0cb-3e83d7b4cf4c" path="/var/lib/kubelet/pods/86239374-2333-40f4-b0cb-3e83d7b4cf4c/volumes"
Oct 30 22:27:34.775341 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:27:34.775299    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:27:46.313133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.313088    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-75jl9"]
Oct 30 22:27:46.313550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.313425    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2d3965d6-4b34-4083-9b03-9937e901c20b" containerName="authorino"
Oct 30 22:27:46.313550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.313438    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3965d6-4b34-4083-9b03-9937e901c20b" containerName="authorino"
Oct 30 22:27:46.313550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.313465    5060 cpu_manager.go:414] "containerMap: removing container" podUID="86239374-2333-40f4-b0cb-3e83d7b4cf4c" containerName="authorino"
Oct 30 22:27:46.313550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.313470    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86239374-2333-40f4-b0cb-3e83d7b4cf4c" containerName="authorino"
Oct 30 22:27:46.313550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.313544    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="86239374-2333-40f4-b0cb-3e83d7b4cf4c" containerName="authorino"
Oct 30 22:27:46.313715 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.313554    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3965d6-4b34-4083-9b03-9937e901c20b" containerName="authorino"
Oct 30 22:27:46.317643 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.317626    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-75jl9"
Oct 30 22:27:46.328322 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.328296    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-75jl9"]
Oct 30 22:27:46.362101 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.362070    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 22:27:46.362238 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.362103    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 22:27:46.362238 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.362110    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 30 22:27:46.362319 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.362237    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-nlj7p\""
Oct 30 22:27:46.389770 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.389740    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90651480-9980-4318-a935-3cde92a8ef96-data\") pod \"seaweedfs-78bb9c6ff8-75jl9\" (UID: \"90651480-9980-4318-a935-3cde92a8ef96\") " pod="kserve/seaweedfs-78bb9c6ff8-75jl9"
Oct 30 22:27:46.389908 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.389779    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnvzg\" (UniqueName: \"kubernetes.io/projected/90651480-9980-4318-a935-3cde92a8ef96-kube-api-access-vnvzg\") pod \"seaweedfs-78bb9c6ff8-75jl9\" (UID: \"90651480-9980-4318-a935-3cde92a8ef96\") " pod="kserve/seaweedfs-78bb9c6ff8-75jl9"
Oct 30 22:27:46.490681 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.490649    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90651480-9980-4318-a935-3cde92a8ef96-data\") pod \"seaweedfs-78bb9c6ff8-75jl9\" (UID: \"90651480-9980-4318-a935-3cde92a8ef96\") " pod="kserve/seaweedfs-78bb9c6ff8-75jl9"
Oct 30 22:27:46.490844 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.490689    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvzg\" (UniqueName: \"kubernetes.io/projected/90651480-9980-4318-a935-3cde92a8ef96-kube-api-access-vnvzg\") pod \"seaweedfs-78bb9c6ff8-75jl9\" (UID: \"90651480-9980-4318-a935-3cde92a8ef96\") " pod="kserve/seaweedfs-78bb9c6ff8-75jl9"
Oct 30 22:27:46.491023 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.491005    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90651480-9980-4318-a935-3cde92a8ef96-data\") pod \"seaweedfs-78bb9c6ff8-75jl9\" (UID: \"90651480-9980-4318-a935-3cde92a8ef96\") " pod="kserve/seaweedfs-78bb9c6ff8-75jl9"
Oct 30 22:27:46.503244 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.503220    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvzg\" (UniqueName: \"kubernetes.io/projected/90651480-9980-4318-a935-3cde92a8ef96-kube-api-access-vnvzg\") pod \"seaweedfs-78bb9c6ff8-75jl9\" (UID: \"90651480-9980-4318-a935-3cde92a8ef96\") " pod="kserve/seaweedfs-78bb9c6ff8-75jl9"
Oct 30 22:27:46.630295 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.630220    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-75jl9"
Oct 30 22:27:46.909404 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:46.909382    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-75jl9"]
Oct 30 22:27:46.911258 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:27:46.911228    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90651480_9980_4318_a935_3cde92a8ef96.slice/crio-87c63cbd18fc5962841fe3ad72f36e1263a4779b65e9b3ae0b3f6d02b13f885e WatchSource:0}: Error finding container 87c63cbd18fc5962841fe3ad72f36e1263a4779b65e9b3ae0b3f6d02b13f885e: Status 404 returned error can't find the container with id 87c63cbd18fc5962841fe3ad72f36e1263a4779b65e9b3ae0b3f6d02b13f885e
Oct 30 22:27:47.628130 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:47.628089    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-75jl9" event={"ID":"90651480-9980-4318-a935-3cde92a8ef96","Type":"ContainerStarted","Data":"87c63cbd18fc5962841fe3ad72f36e1263a4779b65e9b3ae0b3f6d02b13f885e"}
Oct 30 22:27:49.638400 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:49.638361    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-75jl9" event={"ID":"90651480-9980-4318-a935-3cde92a8ef96","Type":"ContainerStarted","Data":"ec2a8487500de344551aa483fb923e7d6a1066e253a78355cf491c8c9b49664e"}
Oct 30 22:27:49.638810 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:49.638492    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-78bb9c6ff8-75jl9"
Oct 30 22:27:49.660335 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:49.660277    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-78bb9c6ff8-75jl9" podStartSLOduration=1.046625247 podStartE2EDuration="3.660263149s" totalImagesPullingTime="2.613637902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:49.659228632 +0000 UTC m=+550.478966454" watchObservedRunningTime="2026-10-30 22:27:49.660263149 +0000 UTC m=+550.480000965"
Oct 30 22:27:55.645539 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:27:55.645488    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-78bb9c6ff8-75jl9"
Oct 30 22:28:04.774672 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:28:04.774635    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:28:34.774836 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:28:34.774799    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:28:39.332056 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:28:39.332028    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:28:39.333677 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:28:39.333652    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:29:04.775046 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:29:04.775009    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:29:08.290237 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.290202    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-56ddc4c448-mzrn8"]
Oct 30 22:29:08.294494 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.294463    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-mzrn8"
Oct 30 22:29:08.300354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.300326    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-mzrn8"]
Oct 30 22:29:08.304335 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.304313    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-sk9q2\""
Oct 30 22:29:08.304451 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.304435    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Oct 30 22:29:08.356470 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.356431    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b18042-569f-4065-b5fe-17396e93712b-tls-certs\") pod \"model-serving-api-56ddc4c448-mzrn8\" (UID: \"a4b18042-569f-4065-b5fe-17396e93712b\") " pod="kserve/model-serving-api-56ddc4c448-mzrn8"
Oct 30 22:29:08.356696 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.356676    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2rg\" (UniqueName: \"kubernetes.io/projected/a4b18042-569f-4065-b5fe-17396e93712b-kube-api-access-zg2rg\") pod \"model-serving-api-56ddc4c448-mzrn8\" (UID: \"a4b18042-569f-4065-b5fe-17396e93712b\") " pod="kserve/model-serving-api-56ddc4c448-mzrn8"
Oct 30 22:29:08.457271 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.457236    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b18042-569f-4065-b5fe-17396e93712b-tls-certs\") pod \"model-serving-api-56ddc4c448-mzrn8\" (UID: \"a4b18042-569f-4065-b5fe-17396e93712b\") " pod="kserve/model-serving-api-56ddc4c448-mzrn8"
Oct 30 22:29:08.457446 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.457312    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2rg\" (UniqueName: \"kubernetes.io/projected/a4b18042-569f-4065-b5fe-17396e93712b-kube-api-access-zg2rg\") pod \"model-serving-api-56ddc4c448-mzrn8\" (UID: \"a4b18042-569f-4065-b5fe-17396e93712b\") " pod="kserve/model-serving-api-56ddc4c448-mzrn8"
Oct 30 22:29:08.459610 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.459589    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b18042-569f-4065-b5fe-17396e93712b-tls-certs\") pod \"model-serving-api-56ddc4c448-mzrn8\" (UID: \"a4b18042-569f-4065-b5fe-17396e93712b\") " pod="kserve/model-serving-api-56ddc4c448-mzrn8"
Oct 30 22:29:08.470067 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.470044    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2rg\" (UniqueName: \"kubernetes.io/projected/a4b18042-569f-4065-b5fe-17396e93712b-kube-api-access-zg2rg\") pod \"model-serving-api-56ddc4c448-mzrn8\" (UID: \"a4b18042-569f-4065-b5fe-17396e93712b\") " pod="kserve/model-serving-api-56ddc4c448-mzrn8"
Oct 30 22:29:08.609702 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.609612    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-mzrn8"
Oct 30 22:29:08.882183 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.882101    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-mzrn8"]
Oct 30 22:29:08.885137 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:29:08.885102    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b18042_569f_4065_b5fe_17396e93712b.slice/crio-687a1719bee4acca8711f42efc30c3679291872d86bba9469b2674b71102a82a WatchSource:0}: Error finding container 687a1719bee4acca8711f42efc30c3679291872d86bba9469b2674b71102a82a: Status 404 returned error can't find the container with id 687a1719bee4acca8711f42efc30c3679291872d86bba9469b2674b71102a82a
Oct 30 22:29:08.964596 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:08.964561    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-mzrn8" event={"ID":"a4b18042-569f-4065-b5fe-17396e93712b","Type":"ContainerStarted","Data":"687a1719bee4acca8711f42efc30c3679291872d86bba9469b2674b71102a82a"}
Oct 30 22:29:11.979773 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:11.979740    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-mzrn8" event={"ID":"a4b18042-569f-4065-b5fe-17396e93712b","Type":"ContainerStarted","Data":"cc16147be832e85d08b70f6d4e6a2e235d48495b44dc03f4493f6047e8432813"}
Oct 30 22:29:11.980174 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:11.979803    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-56ddc4c448-mzrn8"
Oct 30 22:29:12.000829 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:12.000785    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/model-serving-api-56ddc4c448-mzrn8" podStartSLOduration=1.718869886 podStartE2EDuration="4.00077275s" totalImagesPullingTime="2.281902864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:11.99963289 +0000 UTC m=+632.819370748" watchObservedRunningTime="2026-10-30 22:29:12.00077275 +0000 UTC m=+632.820510569"
Oct 30 22:29:22.988260 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:22.988232    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-56ddc4c448-mzrn8"
Oct 30 22:29:34.777329 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:29:34.777278    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:29:34.932145 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:34.932110    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-zx9kx"]
Oct 30 22:29:34.935738 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:34.935716    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zx9kx"
Oct 30 22:29:34.940082 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:34.940052    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 22:29:34.946424 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:34.946400    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zx9kx"]
Oct 30 22:29:34.981015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:34.980974    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0ceeca-1486-49b2-ac12-0f17cf399052-model-cache\") pod \"s3-init-zx9kx\" (UID: \"cc0ceeca-1486-49b2-ac12-0f17cf399052\") " pod="kserve/s3-init-zx9kx"
Oct 30 22:29:34.981015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:34.981019    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cksbj\" (UniqueName: \"kubernetes.io/projected/cc0ceeca-1486-49b2-ac12-0f17cf399052-kube-api-access-cksbj\") pod \"s3-init-zx9kx\" (UID: \"cc0ceeca-1486-49b2-ac12-0f17cf399052\") " pod="kserve/s3-init-zx9kx"
Oct 30 22:29:35.082246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:35.082151    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0ceeca-1486-49b2-ac12-0f17cf399052-model-cache\") pod \"s3-init-zx9kx\" (UID: \"cc0ceeca-1486-49b2-ac12-0f17cf399052\") " pod="kserve/s3-init-zx9kx"
Oct 30 22:29:35.082246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:35.082193    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksbj\" (UniqueName: \"kubernetes.io/projected/cc0ceeca-1486-49b2-ac12-0f17cf399052-kube-api-access-cksbj\") pod \"s3-init-zx9kx\" (UID: \"cc0ceeca-1486-49b2-ac12-0f17cf399052\") " pod="kserve/s3-init-zx9kx"
Oct 30 22:29:35.082632 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:35.082608    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0ceeca-1486-49b2-ac12-0f17cf399052-model-cache\") pod \"s3-init-zx9kx\" (UID: \"cc0ceeca-1486-49b2-ac12-0f17cf399052\") " pod="kserve/s3-init-zx9kx"
Oct 30 22:29:35.095444 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:35.095418    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksbj\" (UniqueName: \"kubernetes.io/projected/cc0ceeca-1486-49b2-ac12-0f17cf399052-kube-api-access-cksbj\") pod \"s3-init-zx9kx\" (UID: \"cc0ceeca-1486-49b2-ac12-0f17cf399052\") " pod="kserve/s3-init-zx9kx"
Oct 30 22:29:35.249772 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:35.249735    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zx9kx"
Oct 30 22:29:35.528561 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:35.528529    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zx9kx"]
Oct 30 22:29:35.532340 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:29:35.532311    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc0ceeca_1486_49b2_ac12_0f17cf399052.slice/crio-a806bbec9d349ab5bc548f753ae14ea8627b3e4f0a8456d3855559cfdc19dd78 WatchSource:0}: Error finding container a806bbec9d349ab5bc548f753ae14ea8627b3e4f0a8456d3855559cfdc19dd78: Status 404 returned error can't find the container with id a806bbec9d349ab5bc548f753ae14ea8627b3e4f0a8456d3855559cfdc19dd78
Oct 30 22:29:36.081467 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:36.081433    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zx9kx" event={"ID":"cc0ceeca-1486-49b2-ac12-0f17cf399052","Type":"ContainerStarted","Data":"a806bbec9d349ab5bc548f753ae14ea8627b3e4f0a8456d3855559cfdc19dd78"}
Oct 30 22:29:41.693385 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:41.693361    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 22:29:42.109128 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:42.109086    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zx9kx" event={"ID":"cc0ceeca-1486-49b2-ac12-0f17cf399052","Type":"ContainerStarted","Data":"256566f0388f31fd258e19d08baaf272951266a10f074576bdbc115c4cfd9aee"}
Oct 30 22:29:44.125395 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:44.125358    5060 generic.go:356] "Generic (PLEG): container finished" podID="cc0ceeca-1486-49b2-ac12-0f17cf399052" containerID="256566f0388f31fd258e19d08baaf272951266a10f074576bdbc115c4cfd9aee" exitCode=0
Oct 30 22:29:44.125788 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:44.125427    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zx9kx" event={"ID":"cc0ceeca-1486-49b2-ac12-0f17cf399052","Type":"ContainerDied","Data":"256566f0388f31fd258e19d08baaf272951266a10f074576bdbc115c4cfd9aee"}
Oct 30 22:29:49.152153 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:49.152121    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zx9kx" event={"ID":"cc0ceeca-1486-49b2-ac12-0f17cf399052","Type":"ContainerStarted","Data":"662f20a79a036f48c2de280e4c29189a2761999d3ed8ff1c9c52e04dc647b073"}
Oct 30 22:29:49.173010 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:29:49.172957    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-zx9kx" podStartSLOduration=3.050380053 podStartE2EDuration="15.17294299s" totalImagesPullingTime="10.493003246s" totalInitContainerRuntime="1.629559691s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:49.172277191 +0000 UTC m=+669.992015012" watchObservedRunningTime="2026-10-30 22:29:49.17294299 +0000 UTC m=+669.992680809"
Oct 30 22:30:04.775241 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:30:04.775131    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:30:34.774660 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:30:34.774626    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:30:54.434841 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:54.434807    5060 generic.go:356] "Generic (PLEG): container finished" podID="cc0ceeca-1486-49b2-ac12-0f17cf399052" containerID="662f20a79a036f48c2de280e4c29189a2761999d3ed8ff1c9c52e04dc647b073" exitCode=0
Oct 30 22:30:54.435289 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:54.434866    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zx9kx" event={"ID":"cc0ceeca-1486-49b2-ac12-0f17cf399052","Type":"ContainerDied","Data":"662f20a79a036f48c2de280e4c29189a2761999d3ed8ff1c9c52e04dc647b073"}
Oct 30 22:30:55.562490 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:55.562466    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zx9kx"
Oct 30 22:30:55.632137 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:55.632106    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0ceeca-1486-49b2-ac12-0f17cf399052-kube-api-access-cksbj\" (UniqueName: \"kubernetes.io/projected/cc0ceeca-1486-49b2-ac12-0f17cf399052-kube-api-access-cksbj\") pod \"cc0ceeca-1486-49b2-ac12-0f17cf399052\" (UID: \"cc0ceeca-1486-49b2-ac12-0f17cf399052\") "
Oct 30 22:30:55.632300 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:55.632166    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0ceeca-1486-49b2-ac12-0f17cf399052-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0ceeca-1486-49b2-ac12-0f17cf399052-model-cache\") pod \"cc0ceeca-1486-49b2-ac12-0f17cf399052\" (UID: \"cc0ceeca-1486-49b2-ac12-0f17cf399052\") "
Oct 30 22:30:55.634256 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:55.634230    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0ceeca-1486-49b2-ac12-0f17cf399052-kube-api-access-cksbj" pod "cc0ceeca-1486-49b2-ac12-0f17cf399052" (UID: "cc0ceeca-1486-49b2-ac12-0f17cf399052"). InnerVolumeSpecName "kube-api-access-cksbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:55.650417 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:55.650394    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0ceeca-1486-49b2-ac12-0f17cf399052-model-cache" pod "cc0ceeca-1486-49b2-ac12-0f17cf399052" (UID: "cc0ceeca-1486-49b2-ac12-0f17cf399052"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:55.733414 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:55.733390    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cksbj\" (UniqueName: \"kubernetes.io/projected/cc0ceeca-1486-49b2-ac12-0f17cf399052-kube-api-access-cksbj\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:55.733414 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:55.733413    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0ceeca-1486-49b2-ac12-0f17cf399052-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:56.445117 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:56.445086    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zx9kx" event={"ID":"cc0ceeca-1486-49b2-ac12-0f17cf399052","Type":"ContainerDied","Data":"a806bbec9d349ab5bc548f753ae14ea8627b3e4f0a8456d3855559cfdc19dd78"}
Oct 30 22:30:56.445262 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:56.445126    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a806bbec9d349ab5bc548f753ae14ea8627b3e4f0a8456d3855559cfdc19dd78"
Oct 30 22:30:56.445262 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:30:56.445101    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zx9kx"
Oct 30 22:31:04.777566 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:31:04.777526    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:31:25.093584 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.093495    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"]
Oct 30 22:31:25.094019 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.093850    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cc0ceeca-1486-49b2-ac12-0f17cf399052" containerName="download-hf-model"
Oct 30 22:31:25.094019 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.093861    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0ceeca-1486-49b2-ac12-0f17cf399052" containerName="download-hf-model"
Oct 30 22:31:25.094019 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.093870    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cc0ceeca-1486-49b2-ac12-0f17cf399052" containerName="s3-init"
Oct 30 22:31:25.094019 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.093875    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0ceeca-1486-49b2-ac12-0f17cf399052" containerName="s3-init"
Oct 30 22:31:25.094019 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.093934    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0ceeca-1486-49b2-ac12-0f17cf399052" containerName="s3-init"
Oct 30 22:31:25.096969 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.096950    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.105480 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.105454    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"cpu-accel-preset-kserve-self-signed-certs\""
Oct 30 22:31:25.105625 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.105526    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"seaweedfs-s3-creds\""
Oct 30 22:31:25.105625 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.105548    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"openshift-service-ca.crt\""
Oct 30 22:31:25.105625 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.105601    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"kube-root-ca.crt\""
Oct 30 22:31:25.105765 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.105739    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"default-dockercfg-mxhvr\""
Oct 30 22:31:25.113304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.113282    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"]
Oct 30 22:31:25.197918 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.197880    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-home\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.198085 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.197928    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-model-cache\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.198085 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.198061    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-kserve-provision-location\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.198156 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.198111    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-tmp-dir\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.198193 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.198155    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7895\" (UniqueName: \"kubernetes.io/projected/bd9b8919-7810-4429-9bec-261e40d2b761-kube-api-access-w7895\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.198193 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.198176    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-dshm\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.198280 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.198240    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd9b8919-7810-4429-9bec-261e40d2b761-tls-certs\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.299599 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.299568    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-kserve-provision-location\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.299804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.299620    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-tmp-dir\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.299804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.299644    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7895\" (UniqueName: \"kubernetes.io/projected/bd9b8919-7810-4429-9bec-261e40d2b761-kube-api-access-w7895\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.299804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.299664    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-dshm\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.299804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.299683    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd9b8919-7810-4429-9bec-261e40d2b761-tls-certs\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.299804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.299726    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-home\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.299804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.299759    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-model-cache\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.300107 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.300044    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-kserve-provision-location\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.300156 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.300135    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-tmp-dir\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.300191 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.300181    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-model-cache\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.300260 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.300240    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-home\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.301934 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.301908    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-dshm\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.302159 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.302143    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd9b8919-7810-4429-9bec-261e40d2b761-tls-certs\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.312905 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.312879    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7895\" (UniqueName: \"kubernetes.io/projected/bd9b8919-7810-4429-9bec-261e40d2b761-kube-api-access-w7895\") pod \"cpu-accel-preset-kserve-5bf588fdc9-7fhs5\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.411011 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.410923    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:31:25.692307 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:25.692280    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"]
Oct 30 22:31:25.694737 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:31:25.694708    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9b8919_7810_4429_9bec_261e40d2b761.slice/crio-61a6912d48d5be2077f79031f5928f568a19895bb53a73548db976d178244dba WatchSource:0}: Error finding container 61a6912d48d5be2077f79031f5928f568a19895bb53a73548db976d178244dba: Status 404 returned error can't find the container with id 61a6912d48d5be2077f79031f5928f568a19895bb53a73548db976d178244dba
Oct 30 22:31:26.448015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.447981    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"]
Oct 30 22:31:26.452127 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.452100    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.456946 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.456924    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Oct 30 22:31:26.461904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.461745    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Oct 30 22:31:26.461904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.461835    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-q6gwn\""
Oct 30 22:31:26.461904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.461876    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Oct 30 22:31:26.464548 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.464523    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"]
Oct 30 22:31:26.575720 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.575686    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5" event={"ID":"bd9b8919-7810-4429-9bec-261e40d2b761","Type":"ContainerStarted","Data":"15b0149dddabc4dbd89fc9f60f85240c0ae8e3a736d800b249b4372b02939842"}
Oct 30 22:31:26.575720 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.575723    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5" event={"ID":"bd9b8919-7810-4429-9bec-261e40d2b761","Type":"ContainerStarted","Data":"61a6912d48d5be2077f79031f5928f568a19895bb53a73548db976d178244dba"}
Oct 30 22:31:26.609696 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.609663    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.609696 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.609705    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.609918 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.609820    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.609918 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.609859    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.609918 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.609878    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs6fh\" (UniqueName: \"kubernetes.io/projected/e1afe156-4d3d-45c8-8a49-081a66ba4e10-kube-api-access-gs6fh\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.610027 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.609928    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.710397 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.710314    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.710574 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.710548    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.710652 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.710592    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs6fh\" (UniqueName: \"kubernetes.io/projected/e1afe156-4d3d-45c8-8a49-081a66ba4e10-kube-api-access-gs6fh\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.710652 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.710625    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.710751 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.710688    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.710751 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.710717    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.710751 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.710719    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.710947 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.710929    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.711007 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.710980    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.712863 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.712843    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.713035 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.713018    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.723448 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.723429    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs6fh\" (UniqueName: \"kubernetes.io/projected/e1afe156-4d3d-45c8-8a49-081a66ba4e10-kube-api-access-gs6fh\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-qmnlf\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:26.770378 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:26.770348    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:27.060572 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:27.060542    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"]
Oct 30 22:31:27.065232 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:31:27.065203    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1afe156_4d3d_45c8_8a49_081a66ba4e10.slice/crio-84d8a1aeb0726d72ac20fcc9ecfca286863d0109218f080dc38f65abbbea462e WatchSource:0}: Error finding container 84d8a1aeb0726d72ac20fcc9ecfca286863d0109218f080dc38f65abbbea462e: Status 404 returned error can't find the container with id 84d8a1aeb0726d72ac20fcc9ecfca286863d0109218f080dc38f65abbbea462e
Oct 30 22:31:27.067884 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:27.067861    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:31:27.581660 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:27.581625    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf" event={"ID":"e1afe156-4d3d-45c8-8a49-081a66ba4e10","Type":"ContainerStarted","Data":"84d8a1aeb0726d72ac20fcc9ecfca286863d0109218f080dc38f65abbbea462e"}
Oct 30 22:31:29.591586 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:29.591550    5060 generic.go:356] "Generic (PLEG): container finished" podID="bd9b8919-7810-4429-9bec-261e40d2b761" containerID="15b0149dddabc4dbd89fc9f60f85240c0ae8e3a736d800b249b4372b02939842" exitCode=0
Oct 30 22:31:29.592092 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:29.591627    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5" event={"ID":"bd9b8919-7810-4429-9bec-261e40d2b761","Type":"ContainerDied","Data":"15b0149dddabc4dbd89fc9f60f85240c0ae8e3a736d800b249b4372b02939842"}
Oct 30 22:31:29.593300 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:29.593277    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf" event={"ID":"e1afe156-4d3d-45c8-8a49-081a66ba4e10","Type":"ContainerStarted","Data":"d8a24aed5891a85dca1b66b593af0520cffc8caed2bd3799692cb27ee6cfc74e"}
Oct 30 22:31:29.642123 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:29.642061    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf" podStartSLOduration=1.668818134 podStartE2EDuration="3.642045002s" totalImagesPullingTime="1.973226868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:29.640687498 +0000 UTC m=+770.460425318" watchObservedRunningTime="2026-10-30 22:31:29.642045002 +0000 UTC m=+770.461782820"
Oct 30 22:31:29.771493 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:29.771451    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:34.783010 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:31:34.782944    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:31:36.771473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:36.771432    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:36.789434 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:36.789396    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:37.653715 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:37.653635    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:31:40.166468 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:31:40.166393    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"]
Oct 30 22:32:02.516520 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:02.516101    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"]
Oct 30 22:32:02.529193 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:32:02.529155    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="d8a24aed5891a85dca1b66b593af0520cffc8caed2bd3799692cb27ee6cfc74e" cmd=["/bin/sleep","15"]
Oct 30 22:32:02.529367 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:02.529274    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:32:02.529367 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:32:02.529294    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf" podUID="e1afe156-4d3d-45c8-8a49-081a66ba4e10" containerName="main" containerID="cri-o://d8a24aed5891a85dca1b66b593af0520cffc8caed2bd3799692cb27ee6cfc74e"
Oct 30 22:32:02.529367 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:02.529331    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf" podUID="e1afe156-4d3d-45c8-8a49-081a66ba4e10" containerName="main" containerID="cri-o://d8a24aed5891a85dca1b66b593af0520cffc8caed2bd3799692cb27ee6cfc74e" gracePeriod=60
Oct 30 22:32:02.638083 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:02.638048    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf" podUID="e1afe156-4d3d-45c8-8a49-081a66ba4e10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Oct 30 22:32:02.771558 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:02.771482    5060 generic.go:356] "Generic (PLEG): container finished" podID="e1afe156-4d3d-45c8-8a49-081a66ba4e10" containerID="d8a24aed5891a85dca1b66b593af0520cffc8caed2bd3799692cb27ee6cfc74e" exitCode=0
Oct 30 22:32:02.771703 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:02.771551    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf" event={"ID":"e1afe156-4d3d-45c8-8a49-081a66ba4e10","Type":"ContainerDied","Data":"d8a24aed5891a85dca1b66b593af0520cffc8caed2bd3799692cb27ee6cfc74e"}
Oct 30 22:32:02.901446 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:02.901420    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:32:03.059283 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.059187    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tls-certs\") pod \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") "
Oct 30 22:32:03.059283 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.059250    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1afe156-4d3d-45c8-8a49-081a66ba4e10-kube-api-access-gs6fh\" (UniqueName: \"kubernetes.io/projected/e1afe156-4d3d-45c8-8a49-081a66ba4e10-kube-api-access-gs6fh\") pod \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") "
Oct 30 22:32:03.059283 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.059282    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-home\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-home\") pod \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") "
Oct 30 22:32:03.059589 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.059350    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tmp-dir\") pod \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") "
Oct 30 22:32:03.059589 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.059378    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-dshm\") pod \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") "
Oct 30 22:32:03.059589 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.059422    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-model-cache\") pod \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\" (UID: \"e1afe156-4d3d-45c8-8a49-081a66ba4e10\") "
Oct 30 22:32:03.059888 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.059836    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tmp-dir" pod "e1afe156-4d3d-45c8-8a49-081a66ba4e10" (UID: "e1afe156-4d3d-45c8-8a49-081a66ba4e10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:03.060016 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.059879    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-home" pod "e1afe156-4d3d-45c8-8a49-081a66ba4e10" (UID: "e1afe156-4d3d-45c8-8a49-081a66ba4e10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:03.060016 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.059996    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-model-cache" pod "e1afe156-4d3d-45c8-8a49-081a66ba4e10" (UID: "e1afe156-4d3d-45c8-8a49-081a66ba4e10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:03.062172 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.062137    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tls-certs" pod "e1afe156-4d3d-45c8-8a49-081a66ba4e10" (UID: "e1afe156-4d3d-45c8-8a49-081a66ba4e10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:03.062172 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.062156    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-dshm" pod "e1afe156-4d3d-45c8-8a49-081a66ba4e10" (UID: "e1afe156-4d3d-45c8-8a49-081a66ba4e10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:03.062322 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.062249    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1afe156-4d3d-45c8-8a49-081a66ba4e10-kube-api-access-gs6fh" pod "e1afe156-4d3d-45c8-8a49-081a66ba4e10" (UID: "e1afe156-4d3d-45c8-8a49-081a66ba4e10"). InnerVolumeSpecName "kube-api-access-gs6fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:03.160632 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.160593    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:03.160632 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.160631    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:03.160835 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.160643    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:03.160835 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.160659    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1afe156-4d3d-45c8-8a49-081a66ba4e10-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:03.160835 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.160674    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs6fh\" (UniqueName: \"kubernetes.io/projected/e1afe156-4d3d-45c8-8a49-081a66ba4e10-kube-api-access-gs6fh\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:03.160835 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.160686    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1afe156-4d3d-45c8-8a49-081a66ba4e10-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:03.779080 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.779045    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf" event={"ID":"e1afe156-4d3d-45c8-8a49-081a66ba4e10","Type":"ContainerDied","Data":"84d8a1aeb0726d72ac20fcc9ecfca286863d0109218f080dc38f65abbbea462e"}
Oct 30 22:32:03.779080 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.779076    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"
Oct 30 22:32:03.779659 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.779113    5060 scope.go:122] "RemoveContainer" containerID="d8a24aed5891a85dca1b66b593af0520cffc8caed2bd3799692cb27ee6cfc74e"
Oct 30 22:32:03.799170 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.799135    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"]
Oct 30 22:32:03.804613 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:03.804580    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-qmnlf"]
Oct 30 22:32:04.783525 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:32:04.783477    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:32:05.295431 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:05.295393    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1afe156-4d3d-45c8-8a49-081a66ba4e10" path="/var/lib/kubelet/pods/e1afe156-4d3d-45c8-8a49-081a66ba4e10/volumes"
Oct 30 22:32:17.857131 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:17.857090    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5" event={"ID":"bd9b8919-7810-4429-9bec-261e40d2b761","Type":"ContainerStarted","Data":"84271170a4486b1ade33b6c25b931891d79a454e8dd066e9b52aede4aac89e64"}
Oct 30 22:32:17.887444 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:17.887337    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5" podStartSLOduration=1.159391472 podStartE2EDuration="52.887318439s" totalImagesPullingTime="48.040578614s" totalInitContainerRuntime="3.687348353s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:17.884325482 +0000 UTC m=+818.704063302" watchObservedRunningTime="2026-10-30 22:32:17.887318439 +0000 UTC m=+818.707056261"
Oct 30 22:32:18.411571 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:18.411499    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:32:32.877094 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:32.877057    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5" podUID="bd9b8919-7810-4429-9bec-261e40d2b761" containerName="main" containerID="cri-o://84271170a4486b1ade33b6c25b931891d79a454e8dd066e9b52aede4aac89e64" gracePeriod=45
Oct 30 22:32:34.775010 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:32:34.774971    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:32:54.501010 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.500931    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"]
Oct 30 22:32:54.501500 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.501345    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e1afe156-4d3d-45c8-8a49-081a66ba4e10" containerName="main"
Oct 30 22:32:54.501500 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.501362    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1afe156-4d3d-45c8-8a49-081a66ba4e10" containerName="main"
Oct 30 22:32:54.501500 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.501466    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1afe156-4d3d-45c8-8a49-081a66ba4e10" containerName="main"
Oct 30 22:32:54.503561 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.503534    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.510307 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.510278    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Oct 30 22:32:54.510446 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.510328    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-qnqtx\""
Oct 30 22:32:54.510446 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.510340    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Oct 30 22:32:54.510446 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.510331    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Oct 30 22:32:54.517227 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.517188    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"]
Oct 30 22:32:54.619589 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.619548    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dprjb\" (UniqueName: \"kubernetes.io/projected/7519043f-fade-4542-96ff-efabc098ecac-kube-api-access-dprjb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.619777 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.619599    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.619777 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.619636    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.619777 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.619691    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.619901 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.619779    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7519043f-fade-4542-96ff-efabc098ecac-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.619901 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.619822    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.720819 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.720783    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.721001 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.720850    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprjb\" (UniqueName: \"kubernetes.io/projected/7519043f-fade-4542-96ff-efabc098ecac-kube-api-access-dprjb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.721001 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.720880    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.721001 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.720924    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.721213 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.721052    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.721213 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.721119    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7519043f-fade-4542-96ff-efabc098ecac-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.721316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.721290    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.721363 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.721347    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.721435 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.721414    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.723018 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.723000    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.723461 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.723439    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7519043f-fade-4542-96ff-efabc098ecac-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.733950 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.733929    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dprjb\" (UniqueName: \"kubernetes.io/projected/7519043f-fade-4542-96ff-efabc098ecac-kube-api-access-dprjb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:54.837831 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:54.837740    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:32:55.120536 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:55.120488    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"]
Oct 30 22:32:55.122432 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:32:55.122398    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7519043f_fade_4542_96ff_efabc098ecac.slice/crio-13db9f7126f8122f264675c3ce482bb05064f3e8545d6736637d85968a70fd20 WatchSource:0}: Error finding container 13db9f7126f8122f264675c3ce482bb05064f3e8545d6736637d85968a70fd20: Status 404 returned error can't find the container with id 13db9f7126f8122f264675c3ce482bb05064f3e8545d6736637d85968a70fd20
Oct 30 22:32:56.042795 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:56.042760    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk" event={"ID":"7519043f-fade-4542-96ff-efabc098ecac","Type":"ContainerStarted","Data":"cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d"}
Oct 30 22:32:56.043198 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:56.042812    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk" event={"ID":"7519043f-fade-4542-96ff-efabc098ecac","Type":"ContainerStarted","Data":"13db9f7126f8122f264675c3ce482bb05064f3e8545d6736637d85968a70fd20"}
Oct 30 22:32:56.066053 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:56.065991    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk" podStartSLOduration=2.065976367 podStartE2EDuration="2.065976367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:56.064229367 +0000 UTC m=+856.883967189" watchObservedRunningTime="2026-10-30 22:32:56.065976367 +0000 UTC m=+856.885714185"
Oct 30 22:32:56.838304 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:32:56.838267    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:33:04.775953 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:33:04.775905    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:33:04.838426 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:04.838391    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:33:04.851104 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:04.851072    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:33:05.098364 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:05.098281    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:33:18.148807 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.148729    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-5bf588fdc9-7fhs5_bd9b8919-7810-4429-9bec-261e40d2b761/main/0.log"
Oct 30 22:33:18.149189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.149071    5060 generic.go:356] "Generic (PLEG): container finished" podID="bd9b8919-7810-4429-9bec-261e40d2b761" containerID="84271170a4486b1ade33b6c25b931891d79a454e8dd066e9b52aede4aac89e64" exitCode=137
Oct 30 22:33:18.149189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.149114    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5" event={"ID":"bd9b8919-7810-4429-9bec-261e40d2b761","Type":"ContainerDied","Data":"84271170a4486b1ade33b6c25b931891d79a454e8dd066e9b52aede4aac89e64"}
Oct 30 22:33:18.521251 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.521227    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-5bf588fdc9-7fhs5_bd9b8919-7810-4429-9bec-261e40d2b761/main/0.log"
Oct 30 22:33:18.522700 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.522678    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:33:18.523906 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.523887    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9b8919-7810-4429-9bec-261e40d2b761-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd9b8919-7810-4429-9bec-261e40d2b761-tls-certs\") pod \"bd9b8919-7810-4429-9bec-261e40d2b761\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") "
Oct 30 22:33:18.524002 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.523939    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-tmp-dir\") pod \"bd9b8919-7810-4429-9bec-261e40d2b761\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") "
Oct 30 22:33:18.524002 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.523966    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9b8919-7810-4429-9bec-261e40d2b761-kube-api-access-w7895\" (UniqueName: \"kubernetes.io/projected/bd9b8919-7810-4429-9bec-261e40d2b761-kube-api-access-w7895\") pod \"bd9b8919-7810-4429-9bec-261e40d2b761\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") "
Oct 30 22:33:18.524124 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.524009    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-home\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-home\") pod \"bd9b8919-7810-4429-9bec-261e40d2b761\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") "
Oct 30 22:33:18.524124 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.524035    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-kserve-provision-location\") pod \"bd9b8919-7810-4429-9bec-261e40d2b761\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") "
Oct 30 22:33:18.524124 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.524060    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-dshm\") pod \"bd9b8919-7810-4429-9bec-261e40d2b761\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") "
Oct 30 22:33:18.524124 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.524104    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-model-cache\") pod \"bd9b8919-7810-4429-9bec-261e40d2b761\" (UID: \"bd9b8919-7810-4429-9bec-261e40d2b761\") "
Oct 30 22:33:18.524616 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.524465    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-home" pod "bd9b8919-7810-4429-9bec-261e40d2b761" (UID: "bd9b8919-7810-4429-9bec-261e40d2b761"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:18.524734 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.524558    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-model-cache" pod "bd9b8919-7810-4429-9bec-261e40d2b761" (UID: "bd9b8919-7810-4429-9bec-261e40d2b761"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:18.525220 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.525197    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-tmp-dir" pod "bd9b8919-7810-4429-9bec-261e40d2b761" (UID: "bd9b8919-7810-4429-9bec-261e40d2b761"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:18.526804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.526715    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9b8919-7810-4429-9bec-261e40d2b761-kube-api-access-w7895" pod "bd9b8919-7810-4429-9bec-261e40d2b761" (UID: "bd9b8919-7810-4429-9bec-261e40d2b761"). InnerVolumeSpecName "kube-api-access-w7895". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:18.526804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.526784    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9b8919-7810-4429-9bec-261e40d2b761-tls-certs" pod "bd9b8919-7810-4429-9bec-261e40d2b761" (UID: "bd9b8919-7810-4429-9bec-261e40d2b761"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:18.526965 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.526799    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-dshm" pod "bd9b8919-7810-4429-9bec-261e40d2b761" (UID: "bd9b8919-7810-4429-9bec-261e40d2b761"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:18.588577 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.588528    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-kserve-provision-location" pod "bd9b8919-7810-4429-9bec-261e40d2b761" (UID: "bd9b8919-7810-4429-9bec-261e40d2b761"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:18.625543 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.625483    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.625543 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.625529    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd9b8919-7810-4429-9bec-261e40d2b761-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.625543 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.625546    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.625769 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.625560    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7895\" (UniqueName: \"kubernetes.io/projected/bd9b8919-7810-4429-9bec-261e40d2b761-kube-api-access-w7895\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.625769 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.625570    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.625769 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.625578    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.625769 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:18.625587    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd9b8919-7810-4429-9bec-261e40d2b761-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:19.156017 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:19.155985    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-5bf588fdc9-7fhs5_bd9b8919-7810-4429-9bec-261e40d2b761/main/0.log"
Oct 30 22:33:19.156490 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:19.156380    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5" event={"ID":"bd9b8919-7810-4429-9bec-261e40d2b761","Type":"ContainerDied","Data":"61a6912d48d5be2077f79031f5928f568a19895bb53a73548db976d178244dba"}
Oct 30 22:33:19.156490 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:19.156392    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"
Oct 30 22:33:19.156490 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:19.156437    5060 scope.go:122] "RemoveContainer" containerID="84271170a4486b1ade33b6c25b931891d79a454e8dd066e9b52aede4aac89e64"
Oct 30 22:33:19.165317 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:19.165297    5060 scope.go:122] "RemoveContainer" containerID="15b0149dddabc4dbd89fc9f60f85240c0ae8e3a736d800b249b4372b02939842"
Oct 30 22:33:19.182076 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:19.182049    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"]
Oct 30 22:33:19.188630 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:19.188609    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-5bf588fdc9-7fhs5"]
Oct 30 22:33:19.290893 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:19.290862    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9b8919-7810-4429-9bec-261e40d2b761" path="/var/lib/kubelet/pods/bd9b8919-7810-4429-9bec-261e40d2b761/volumes"
Oct 30 22:33:29.826609 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:29.826572    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"]
Oct 30 22:33:29.836622 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:33:29.836583    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d" cmd=["/bin/sleep","15"]
Oct 30 22:33:29.836793 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:29.836637    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:33:29.836793 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:33:29.836651    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk" podUID="7519043f-fade-4542-96ff-efabc098ecac" containerName="main" containerID="cri-o://cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d"
Oct 30 22:33:29.836793 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:29.836692    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk" podUID="7519043f-fade-4542-96ff-efabc098ecac" containerName="main" containerID="cri-o://cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d" gracePeriod=60
Oct 30 22:33:30.088104 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.088000    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk" podUID="7519043f-fade-4542-96ff-efabc098ecac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Oct 30 22:33:30.189942 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.189910    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:33:30.210102 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.210067    5060 generic.go:356] "Generic (PLEG): container finished" podID="7519043f-fade-4542-96ff-efabc098ecac" containerID="cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d" exitCode=0
Oct 30 22:33:30.210280 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.210115    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"
Oct 30 22:33:30.210280 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.210106    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk" event={"ID":"7519043f-fade-4542-96ff-efabc098ecac","Type":"ContainerDied","Data":"cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d"}
Oct 30 22:33:30.210280 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.210235    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk" event={"ID":"7519043f-fade-4542-96ff-efabc098ecac","Type":"ContainerDied","Data":"13db9f7126f8122f264675c3ce482bb05064f3e8545d6736637d85968a70fd20"}
Oct 30 22:33:30.210280 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.210268    5060 scope.go:122] "RemoveContainer" containerID="cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d"
Oct 30 22:33:30.220006 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.219953    5060 scope.go:122] "RemoveContainer" containerID="cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d"
Oct 30 22:33:30.220294 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:33:30.220258    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d\": container with ID starting with cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d not found: ID does not exist" containerID="cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d"
Oct 30 22:33:30.220398 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.220290    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d"} err="failed to get container status \"cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d\": rpc error: code = NotFound desc = could not find container \"cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d\": container with ID starting with cf6621146872caeb107801f99dc91b46768c978667c34c229f6ae67b9e00fe3d not found: ID does not exist"
Oct 30 22:33:30.224686 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.224664    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-tmp-dir\") pod \"7519043f-fade-4542-96ff-efabc098ecac\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") "
Oct 30 22:33:30.224773 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.224712    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7519043f-fade-4542-96ff-efabc098ecac-tls-certs\" (UniqueName: \"kubernetes.io/secret/7519043f-fade-4542-96ff-efabc098ecac-tls-certs\") pod \"7519043f-fade-4542-96ff-efabc098ecac\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") "
Oct 30 22:33:30.224773 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.224739    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-dshm\") pod \"7519043f-fade-4542-96ff-efabc098ecac\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") "
Oct 30 22:33:30.224773 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.224766    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7519043f-fade-4542-96ff-efabc098ecac-kube-api-access-dprjb\" (UniqueName: \"kubernetes.io/projected/7519043f-fade-4542-96ff-efabc098ecac-kube-api-access-dprjb\") pod \"7519043f-fade-4542-96ff-efabc098ecac\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") "
Oct 30 22:33:30.224876 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.224817    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-model-cache\") pod \"7519043f-fade-4542-96ff-efabc098ecac\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") "
Oct 30 22:33:30.224944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.224929    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-home\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-home\") pod \"7519043f-fade-4542-96ff-efabc098ecac\" (UID: \"7519043f-fade-4542-96ff-efabc098ecac\") "
Oct 30 22:33:30.225189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.224933    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-tmp-dir" pod "7519043f-fade-4542-96ff-efabc098ecac" (UID: "7519043f-fade-4542-96ff-efabc098ecac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:30.225189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.225119    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-model-cache" pod "7519043f-fade-4542-96ff-efabc098ecac" (UID: "7519043f-fade-4542-96ff-efabc098ecac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:30.225189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.225133    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-home" pod "7519043f-fade-4542-96ff-efabc098ecac" (UID: "7519043f-fade-4542-96ff-efabc098ecac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:30.228779 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.228755    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7519043f-fade-4542-96ff-efabc098ecac-tls-certs" pod "7519043f-fade-4542-96ff-efabc098ecac" (UID: "7519043f-fade-4542-96ff-efabc098ecac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:30.228870 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.228817    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7519043f-fade-4542-96ff-efabc098ecac-kube-api-access-dprjb" pod "7519043f-fade-4542-96ff-efabc098ecac" (UID: "7519043f-fade-4542-96ff-efabc098ecac"). InnerVolumeSpecName "kube-api-access-dprjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:30.228989 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.228974    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-dshm" pod "7519043f-fade-4542-96ff-efabc098ecac" (UID: "7519043f-fade-4542-96ff-efabc098ecac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:30.326252 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.326210    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7519043f-fade-4542-96ff-efabc098ecac-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:30.326252 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.326244    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:30.326252 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.326257    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dprjb\" (UniqueName: \"kubernetes.io/projected/7519043f-fade-4542-96ff-efabc098ecac-kube-api-access-dprjb\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:30.326503 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.326270    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:30.326503 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.326282    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:30.326503 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.326293    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7519043f-fade-4542-96ff-efabc098ecac-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:30.533863 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.533826    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"]
Oct 30 22:33:30.539009 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:30.538975    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76nbxgk"]
Oct 30 22:33:31.289198 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:31.289164    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7519043f-fade-4542-96ff-efabc098ecac" path="/var/lib/kubelet/pods/7519043f-fade-4542-96ff-efabc098ecac/volumes"
Oct 30 22:33:34.775055 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:33:34.775017    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:33:34.963155 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.963123    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"]
Oct 30 22:33:34.963542 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.963528    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bd9b8919-7810-4429-9bec-261e40d2b761" containerName="main"
Oct 30 22:33:34.963594 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.963544    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9b8919-7810-4429-9bec-261e40d2b761" containerName="main"
Oct 30 22:33:34.963594 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.963564    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bd9b8919-7810-4429-9bec-261e40d2b761" containerName="storage-initializer"
Oct 30 22:33:34.963594 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.963569    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9b8919-7810-4429-9bec-261e40d2b761" containerName="storage-initializer"
Oct 30 22:33:34.963594 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.963575    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7519043f-fade-4542-96ff-efabc098ecac" containerName="main"
Oct 30 22:33:34.963594 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.963580    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7519043f-fade-4542-96ff-efabc098ecac" containerName="main"
Oct 30 22:33:34.963744 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.963646    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9b8919-7810-4429-9bec-261e40d2b761" containerName="main"
Oct 30 22:33:34.963744 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.963657    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7519043f-fade-4542-96ff-efabc098ecac" containerName="main"
Oct 30 22:33:34.965614 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.965597    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:34.970402 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.970368    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"seaweedfs-s3-creds\""
Oct 30 22:33:34.971592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.971573    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"cpu-accel-precedence-kserve-self-signed-certs\""
Oct 30 22:33:34.971862 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.971849    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"default-dockercfg-rdhz5\""
Oct 30 22:33:34.971978 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.971860    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"openshift-service-ca.crt\""
Oct 30 22:33:34.971978 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.971949    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"kube-root-ca.crt\""
Oct 30 22:33:34.979479 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:34.979458    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"]
Oct 30 22:33:35.064048 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.063950    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dzfh\" (UniqueName: \"kubernetes.io/projected/15b5bc12-034a-450c-be74-36e5ec95e170-kube-api-access-9dzfh\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.064048 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.063994    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-model-cache\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.064255 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.064110    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-dshm\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.064255 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.064149    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-home\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.064255 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.064175    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15b5bc12-034a-450c-be74-36e5ec95e170-tls-certs\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.064365 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.064250    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-tmp-dir\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.064365 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.064295    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.165565 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.165497    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dzfh\" (UniqueName: \"kubernetes.io/projected/15b5bc12-034a-450c-be74-36e5ec95e170-kube-api-access-9dzfh\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.165565 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.165565    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-model-cache\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.165821 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.165604    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-dshm\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.165821 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.165626    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-home\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.165821 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.165772    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15b5bc12-034a-450c-be74-36e5ec95e170-tls-certs\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.165821 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.165814    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-tmp-dir\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.166054 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.165872    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.166054 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.166000    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-home\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.166148 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.166077    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-model-cache\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.166182 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.166151    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-tmp-dir\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.166182 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.166171    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.167970 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.167946    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-dshm\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.168211 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.168194    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15b5bc12-034a-450c-be74-36e5ec95e170-tls-certs\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.179271 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.179244    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dzfh\" (UniqueName: \"kubernetes.io/projected/15b5bc12-034a-450c-be74-36e5ec95e170-kube-api-access-9dzfh\") pod \"cpu-accel-precedence-kserve-7489f44cbc-nhvpp\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.278266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.278221    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:33:35.561188 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:35.561157    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"]
Oct 30 22:33:35.563833 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:33:35.563805    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b5bc12_034a_450c_be74_36e5ec95e170.slice/crio-6919521f4954a443b0966c3614c67721e9da48b04e4d187d3471b8724eb5cd3b WatchSource:0}: Error finding container 6919521f4954a443b0966c3614c67721e9da48b04e4d187d3471b8724eb5cd3b: Status 404 returned error can't find the container with id 6919521f4954a443b0966c3614c67721e9da48b04e4d187d3471b8724eb5cd3b
Oct 30 22:33:36.236801 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.236766    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp" event={"ID":"15b5bc12-034a-450c-be74-36e5ec95e170","Type":"ContainerStarted","Data":"9af02433661b8744c9488d399c62854799a4f00fcb09c0acb9db591b88f4c631"}
Oct 30 22:33:36.236801 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.236805    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp" event={"ID":"15b5bc12-034a-450c-be74-36e5ec95e170","Type":"ContainerStarted","Data":"6919521f4954a443b0966c3614c67721e9da48b04e4d187d3471b8724eb5cd3b"}
Oct 30 22:33:36.382152 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.382118    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"]
Oct 30 22:33:36.385912 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.385888    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.407716 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.407686    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"]
Oct 30 22:33:36.470713 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.470689    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-kxprg\""
Oct 30 22:33:36.477131 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.477111    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.477192 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.477141    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.477192 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.477160    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.477310 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.477291    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.477348 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.477320    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.477380 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.477367    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.477452 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.477439    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzvjd\" (UniqueName: \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-kube-api-access-mzvjd\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.477485 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.477469    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.477533 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.477497    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.481343 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.481329    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:33:36.578048 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.577974    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578048 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578014    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578048 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578037    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578331 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578134    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578331 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578213    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578331 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578243    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578331 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578270    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578553 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578340    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzvjd\" (UniqueName: \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-kube-api-access-mzvjd\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578553 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578373    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578553 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578413    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578730 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578581    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578730 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578656    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.578849 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.578802    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.579077 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.579057    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.580388 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.580371    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.580746 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.580728    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.593159 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.593130    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.593271 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.593212    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzvjd\" (UniqueName: \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-kube-api-access-mzvjd\") pod \"openshift-ai-inference-openshift-default-86b669b864-rd99c\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.701526 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.701463    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:36.997615 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:36.997492    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"]
Oct 30 22:33:37.000729 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:33:37.000691    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79eadfc_8ca8_408a_9797_30a9d24dbdfa.slice/crio-86a2654090cf0b83434bf638231a0dc2daeba7b83a122eb2cb73288e5fac5ccd WatchSource:0}: Error finding container 86a2654090cf0b83434bf638231a0dc2daeba7b83a122eb2cb73288e5fac5ccd: Status 404 returned error can't find the container with id 86a2654090cf0b83434bf638231a0dc2daeba7b83a122eb2cb73288e5fac5ccd
Oct 30 22:33:37.244272 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:37.244211    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" event={"ID":"b79eadfc-8ca8-408a-9797-30a9d24dbdfa","Type":"ContainerStarted","Data":"86a2654090cf0b83434bf638231a0dc2daeba7b83a122eb2cb73288e5fac5ccd"}
Oct 30 22:33:39.260217 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:39.260169    5060 generic.go:356] "Generic (PLEG): container finished" podID="15b5bc12-034a-450c-be74-36e5ec95e170" containerID="9af02433661b8744c9488d399c62854799a4f00fcb09c0acb9db591b88f4c631" exitCode=0
Oct 30 22:33:39.260690 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:39.260357    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp" event={"ID":"15b5bc12-034a-450c-be74-36e5ec95e170","Type":"ContainerDied","Data":"9af02433661b8744c9488d399c62854799a4f00fcb09c0acb9db591b88f4c631"}
Oct 30 22:33:39.364089 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:39.364060    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:33:39.366373 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:39.366347    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:33:40.068252 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:40.068203    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"]
Oct 30 22:33:40.495816 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:40.495757    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:33:40.496189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:40.495847    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:33:40.496189 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:40.495887    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:33:41.296088 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:41.296056    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" event={"ID":"b79eadfc-8ca8-408a-9797-30a9d24dbdfa","Type":"ContainerStarted","Data":"3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4"}
Oct 30 22:33:41.320832 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:41.320777    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" podStartSLOduration=1.829190933 podStartE2EDuration="5.320759728s" totalImagesPullingTime="3.491568795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:41.31836638 +0000 UTC m=+902.138104200" watchObservedRunningTime="2026-10-30 22:33:41.320759728 +0000 UTC m=+902.140497548"
Oct 30 22:33:41.702324 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:41.702293    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:41.708824 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:41.708794    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:42.297270 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:42.297234    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:33:42.298471 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:33:42.298449    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:34:04.775712 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:34:04.775674    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:34:07.436370 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:07.436278    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp" event={"ID":"15b5bc12-034a-450c-be74-36e5ec95e170","Type":"ContainerStarted","Data":"e2f69ebc3d11a4f0858f46fcc0eeda3613b95b0181e423107b31e49209f53104"}
Oct 30 22:34:07.468006 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:07.467947    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp" podStartSLOduration=2.024227486 podStartE2EDuration="33.467934349s" totalImagesPullingTime="27.895856111s" totalInitContainerRuntime="3.547850752s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:07.465320422 +0000 UTC m=+928.285058241" watchObservedRunningTime="2026-10-30 22:34:07.467934349 +0000 UTC m=+928.287672205"
Oct 30 22:34:08.278375 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:08.278329    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:34:22.458298 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:22.458187    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp" podUID="15b5bc12-034a-450c-be74-36e5ec95e170" containerName="main" containerID="cri-o://e2f69ebc3d11a4f0858f46fcc0eeda3613b95b0181e423107b31e49209f53104" gracePeriod=45
Oct 30 22:34:24.680239 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:24.680206    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"]
Oct 30 22:34:25.349458 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.349414    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"]
Oct 30 22:34:25.349703 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.349571    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.361114 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.361081    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Oct 30 22:34:25.361311 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.361096    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Oct 30 22:34:25.361409 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.361099    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Oct 30 22:34:25.361409 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.361179    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-6m677\""
Oct 30 22:34:25.420440 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.420405    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.420610 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.420551    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.420664 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.420624    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.420732 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.420707    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vfz8\" (UniqueName: \"kubernetes.io/projected/9f34c11f-4bd5-49fd-a943-9c913afe4605-kube-api-access-4vfz8\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.420816 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.420798    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.420873 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.420830    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f34c11f-4bd5-49fd-a943-9c913afe4605-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.521758 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.521727    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.521957 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.521772    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vfz8\" (UniqueName: \"kubernetes.io/projected/9f34c11f-4bd5-49fd-a943-9c913afe4605-kube-api-access-4vfz8\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.521957 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.521825    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.521957 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.521852    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f34c11f-4bd5-49fd-a943-9c913afe4605-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.521957 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.521879    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.522164 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.521971    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.522219 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.522197    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-home\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.522275 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.522217    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.522398 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.522379    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-model-cache\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.524196 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.524171    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-dshm\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.524289 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.524273    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f34c11f-4bd5-49fd-a943-9c913afe4605-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.535198 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.535178    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vfz8\" (UniqueName: \"kubernetes.io/projected/9f34c11f-4bd5-49fd-a943-9c913afe4605-kube-api-access-4vfz8\") pod \"scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:25.697444 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:25.693615    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:26.018706 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:26.018615    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"]
Oct 30 22:34:26.021344 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:34:26.021319    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f34c11f_4bd5_49fd_a943_9c913afe4605.slice/crio-b189e5558f49fb473d27d26068987fd7e36357cd1a4427fe415f384f75266781 WatchSource:0}: Error finding container b189e5558f49fb473d27d26068987fd7e36357cd1a4427fe415f384f75266781: Status 404 returned error can't find the container with id b189e5558f49fb473d27d26068987fd7e36357cd1a4427fe415f384f75266781
Oct 30 22:34:26.526669 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:26.526627    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv" event={"ID":"9f34c11f-4bd5-49fd-a943-9c913afe4605","Type":"ContainerStarted","Data":"f8c0900034a68850c6e0c96d7c38bf91be222b27aa63e7c002b5c0626d87d7a1"}
Oct 30 22:34:26.526832 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:26.526697    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv" event={"ID":"9f34c11f-4bd5-49fd-a943-9c913afe4605","Type":"ContainerStarted","Data":"b189e5558f49fb473d27d26068987fd7e36357cd1a4427fe415f384f75266781"}
Oct 30 22:34:26.547316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:26.547261    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv" podStartSLOduration=2.547234146 podStartE2EDuration="2.547234146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:26.546259847 +0000 UTC m=+947.365997667" watchObservedRunningTime="2026-10-30 22:34:26.547234146 +0000 UTC m=+947.366971965"
Oct 30 22:34:26.696810 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:26.696778    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:34.564520 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:34.564483    5060 generic.go:356] "Generic (PLEG): container finished" podID="b79eadfc-8ca8-408a-9797-30a9d24dbdfa" containerID="3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4" exitCode=0
Oct 30 22:34:34.564915 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:34.564558    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" event={"ID":"b79eadfc-8ca8-408a-9797-30a9d24dbdfa","Type":"ContainerDied","Data":"3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4"}
Oct 30 22:34:34.564915 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:34.564869    5060 scope.go:122] "RemoveContainer" containerID="3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4"
Oct 30 22:34:34.565916 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:34.565877    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:34.565985 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:34.565970    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:34.566033 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:34.566018    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:34.777288 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:34:34.777141    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:34:35.579001 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:35.578965    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" event={"ID":"b79eadfc-8ca8-408a-9797-30a9d24dbdfa","Type":"ContainerStarted","Data":"a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d"}
Oct 30 22:34:35.697323 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:35.697293    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:35.702162 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:35.702102    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:34:35.702321 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:35.702297    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-rd99c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:34:35.702385 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:35.702362    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" podUID="b79eadfc-8ca8-408a-9797-30a9d24dbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:34:35.709481 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:35.709463    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:36.594227 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:36.594198    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:36.702413 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:36.702385    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-rd99c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:34:36.702586 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:36.702435    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" podUID="b79eadfc-8ca8-408a-9797-30a9d24dbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:34:37.702484 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:37.702448    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-rd99c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:34:37.702972 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:37.702529    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" podUID="b79eadfc-8ca8-408a-9797-30a9d24dbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:34:38.702202 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:38.702165    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-rd99c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:34:38.702380 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:38.702219    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" podUID="b79eadfc-8ca8-408a-9797-30a9d24dbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:34:39.702249 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:39.702214    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-rd99c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 22:34:39.702669 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:39.702269    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" podUID="b79eadfc-8ca8-408a-9797-30a9d24dbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 22:34:40.706409 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:40.706377    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:34:40.706853 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:40.706737    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:34:40.707312 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:40.707294    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 22:34:51.473642 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.473603    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"]
Oct 30 22:34:51.480246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.480222    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.491182 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.491149    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"]
Oct 30 22:34:51.493826 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.493799    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.493963 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.493848    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.494037 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.493983    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.494153 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.494019    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.494257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.494193    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.494257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.494229    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.494257 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.494250    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e16630-c4bf-4d65-b439-4626bb1a890f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.494390 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.494273    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrmmh\" (UniqueName: \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-kube-api-access-wrmmh\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.494390 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.494338    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.523332 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.523303    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"]
Oct 30 22:34:51.525772 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.525754    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.547940 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.547914    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"]
Oct 30 22:34:51.594880 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.594852    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.594880 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.594892    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595114 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.594916    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e16630-c4bf-4d65-b439-4626bb1a890f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595114 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.594940    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrmmh\" (UniqueName: \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-kube-api-access-wrmmh\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595114 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.594967    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.595114 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.594990    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.595114 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595015    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.595114 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595034    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.595114 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595070    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.595492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595197    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595264    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595407    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595448    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.595492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595476    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44629e36-a027-4fac-9839-b268be7f21d7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.595766 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595564    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595766 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595590    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44629e36-a027-4fac-9839-b268be7f21d7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.595766 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595630    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7wt\" (UniqueName: \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-kube-api-access-jx7wt\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.595766 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595732    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e16630-c4bf-4d65-b439-4626bb1a890f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595949 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595761    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595949 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595817    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595949 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595839    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.595949 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.595874    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.596107 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.596029    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.597696 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.597670    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.597968 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.597951    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.608087 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.608064    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrmmh\" (UniqueName: \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-kube-api-access-wrmmh\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.608182 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.608090    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-xdwvk\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.696455 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.696428    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.696455 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.696455    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44629e36-a027-4fac-9839-b268be7f21d7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.696665 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.696477    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44629e36-a027-4fac-9839-b268be7f21d7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.696665 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.696493    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7wt\" (UniqueName: \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-kube-api-access-jx7wt\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.696665 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.696576    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.696665 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.696602    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.696868 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.696742    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.696868 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.696797    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.696994 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.696970    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.697061 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.697016    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.697213 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.697189    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.697326 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.697280    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.697326 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.697292    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44629e36-a027-4fac-9839-b268be7f21d7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.697447 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.697421    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.698790 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.698767    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.698917 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.698899    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44629e36-a027-4fac-9839-b268be7f21d7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.709461 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.709441    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.709673 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.709649    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7wt\" (UniqueName: \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-kube-api-access-jx7wt\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghl2l\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:51.792758 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.792696    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:51.838405 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:51.838371    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:52.078802 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.078780    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"]
Oct 30 22:34:52.081625 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:34:52.081595    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e16630_c4bf_4d65_b439_4626bb1a890f.slice/crio-6f7f0f96a3edb6c9180a309f930b29e8d4bbf41dfa8b3b8f2fc6a1deb56b1d42 WatchSource:0}: Error finding container 6f7f0f96a3edb6c9180a309f930b29e8d4bbf41dfa8b3b8f2fc6a1deb56b1d42: Status 404 returned error can't find the container with id 6f7f0f96a3edb6c9180a309f930b29e8d4bbf41dfa8b3b8f2fc6a1deb56b1d42
Oct 30 22:34:52.084169 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.084139    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:52.084229 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.084198    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:52.084266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.084228    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:52.128345 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.128325    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"]
Oct 30 22:34:52.131395 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:34:52.131369    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44629e36_a027_4fac_9839_b268be7f21d7.slice/crio-b66499111c3fc05c5e0207b4cb28dfa71b799cf9daabc4110d808732c57bb68e WatchSource:0}: Error finding container b66499111c3fc05c5e0207b4cb28dfa71b799cf9daabc4110d808732c57bb68e: Status 404 returned error can't find the container with id b66499111c3fc05c5e0207b4cb28dfa71b799cf9daabc4110d808732c57bb68e
Oct 30 22:34:52.134068 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.134037    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:52.134172 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.134101    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:52.134172 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.134128    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:52.711982 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.711933    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" event={"ID":"44629e36-a027-4fac-9839-b268be7f21d7","Type":"ContainerStarted","Data":"585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35"}
Oct 30 22:34:52.712466 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.712092    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" event={"ID":"44629e36-a027-4fac-9839-b268be7f21d7","Type":"ContainerStarted","Data":"b66499111c3fc05c5e0207b4cb28dfa71b799cf9daabc4110d808732c57bb68e"}
Oct 30 22:34:52.713656 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.713632    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" event={"ID":"39e16630-c4bf-4d65-b439-4626bb1a890f","Type":"ContainerStarted","Data":"b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522"}
Oct 30 22:34:52.713771 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.713670    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" event={"ID":"39e16630-c4bf-4d65-b439-4626bb1a890f","Type":"ContainerStarted","Data":"6f7f0f96a3edb6c9180a309f930b29e8d4bbf41dfa8b3b8f2fc6a1deb56b1d42"}
Oct 30 22:34:52.740706 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.740663    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" podStartSLOduration=1.740652131 podStartE2EDuration="1.740652131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:52.738251481 +0000 UTC m=+973.557989301" watchObservedRunningTime="2026-10-30 22:34:52.740652131 +0000 UTC m=+973.560389947"
Oct 30 22:34:52.795530 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.793669    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:52.838466 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:52.838434    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:53.799531 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:53.798536    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-xdwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 22:34:53.799531 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:53.798602    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 22:34:53.839814 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:53.839779    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:34:53.839980 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:53.839841    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" podUID="44629e36-a027-4fac-9839-b268be7f21d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:34:54.793902 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:54.793864    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-xdwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 22:34:54.794183 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:54.794151    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 22:34:54.839922 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:54.839358    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:34:54.839922 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:54.839426    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" podUID="44629e36-a027-4fac-9839-b268be7f21d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:34:55.793949 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:55.793909    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-xdwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 22:34:55.794224 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:55.794192    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 22:34:55.839117 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:55.839079    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:34:55.839300 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:55.839127    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" podUID="44629e36-a027-4fac-9839-b268be7f21d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:34:56.796934 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:56.796889    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-xdwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 22:34:56.797435 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:56.796964    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 22:34:56.839078 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:56.839045    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:34:56.839245 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:56.839093    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" podUID="44629e36-a027-4fac-9839-b268be7f21d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:34:57.795951 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:57.795909    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-xdwvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 22:34:57.796243 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:57.796208    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 22:34:57.839344 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:57.839305    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghl2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 22:34:57.839757 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:57.839377    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" podUID="44629e36-a027-4fac-9839-b268be7f21d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 22:34:58.482758 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.482697    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" podStartSLOduration=7.482678026 podStartE2EDuration="7.482678026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:52.795865007 +0000 UTC m=+973.615602848" watchObservedRunningTime="2026-10-30 22:34:58.482678026 +0000 UTC m=+979.302415846"
Oct 30 22:34:58.484039 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.484019    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"]
Oct 30 22:34:58.494369 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:34:58.494328    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="f8c0900034a68850c6e0c96d7c38bf91be222b27aa63e7c002b5c0626d87d7a1" cmd=["/bin/sleep","15"]
Oct 30 22:34:58.494540 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.494384    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:34:58.494540 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:34:58.494396    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv" podUID="9f34c11f-4bd5-49fd-a943-9c913afe4605" containerName="main" containerID="cri-o://f8c0900034a68850c6e0c96d7c38bf91be222b27aa63e7c002b5c0626d87d7a1"
Oct 30 22:34:58.494540 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.494445    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv" podUID="9f34c11f-4bd5-49fd-a943-9c913afe4605" containerName="main" containerID="cri-o://f8c0900034a68850c6e0c96d7c38bf91be222b27aa63e7c002b5c0626d87d7a1" gracePeriod=60
Oct 30 22:34:58.583840 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.583785    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv" podUID="9f34c11f-4bd5-49fd-a943-9c913afe4605" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Oct 30 22:34:58.772498 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.772368    5060 generic.go:356] "Generic (PLEG): container finished" podID="9f34c11f-4bd5-49fd-a943-9c913afe4605" containerID="f8c0900034a68850c6e0c96d7c38bf91be222b27aa63e7c002b5c0626d87d7a1" exitCode=0
Oct 30 22:34:58.772498 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.772412    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv" event={"ID":"9f34c11f-4bd5-49fd-a943-9c913afe4605","Type":"ContainerDied","Data":"f8c0900034a68850c6e0c96d7c38bf91be222b27aa63e7c002b5c0626d87d7a1"}
Oct 30 22:34:58.811803 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.810392    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:58.811803 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.811630    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:58.812959 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.812885    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 22:34:58.847066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.847038    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:58.850164 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.850137    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:58.850619 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.850600    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 22:34:58.966499 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:58.964474    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:59.002391 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.002368    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-home\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-home\") pod \"9f34c11f-4bd5-49fd-a943-9c913afe4605\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") "
Oct 30 22:34:59.002539 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.002404    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-dshm\") pod \"9f34c11f-4bd5-49fd-a943-9c913afe4605\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") "
Oct 30 22:34:59.002539 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.002469    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f34c11f-4bd5-49fd-a943-9c913afe4605-kube-api-access-4vfz8\" (UniqueName: \"kubernetes.io/projected/9f34c11f-4bd5-49fd-a943-9c913afe4605-kube-api-access-4vfz8\") pod \"9f34c11f-4bd5-49fd-a943-9c913afe4605\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") "
Oct 30 22:34:59.003301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.002568    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f34c11f-4bd5-49fd-a943-9c913afe4605-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f34c11f-4bd5-49fd-a943-9c913afe4605-tls-certs\") pod \"9f34c11f-4bd5-49fd-a943-9c913afe4605\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") "
Oct 30 22:34:59.003301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.002611    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-tmp-dir\") pod \"9f34c11f-4bd5-49fd-a943-9c913afe4605\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") "
Oct 30 22:34:59.003301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.002646    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-model-cache\") pod \"9f34c11f-4bd5-49fd-a943-9c913afe4605\" (UID: \"9f34c11f-4bd5-49fd-a943-9c913afe4605\") "
Oct 30 22:34:59.003301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.002696    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-home" pod "9f34c11f-4bd5-49fd-a943-9c913afe4605" (UID: "9f34c11f-4bd5-49fd-a943-9c913afe4605"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:59.003301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.002881    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-tmp-dir" pod "9f34c11f-4bd5-49fd-a943-9c913afe4605" (UID: "9f34c11f-4bd5-49fd-a943-9c913afe4605"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:59.003301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.002926    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:59.003301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.002948    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:59.003301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.003053    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-model-cache" pod "9f34c11f-4bd5-49fd-a943-9c913afe4605" (UID: "9f34c11f-4bd5-49fd-a943-9c913afe4605"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:59.005476 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.005448    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-dshm" pod "9f34c11f-4bd5-49fd-a943-9c913afe4605" (UID: "9f34c11f-4bd5-49fd-a943-9c913afe4605"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:59.005609 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.005448    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f34c11f-4bd5-49fd-a943-9c913afe4605-kube-api-access-4vfz8" pod "9f34c11f-4bd5-49fd-a943-9c913afe4605" (UID: "9f34c11f-4bd5-49fd-a943-9c913afe4605"). InnerVolumeSpecName "kube-api-access-4vfz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:59.005926 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.005898    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f34c11f-4bd5-49fd-a943-9c913afe4605-tls-certs" pod "9f34c11f-4bd5-49fd-a943-9c913afe4605" (UID: "9f34c11f-4bd5-49fd-a943-9c913afe4605"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:59.104252 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.104219    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f34c11f-4bd5-49fd-a943-9c913afe4605-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:59.104252 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.104248    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:59.104252 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.104257    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f34c11f-4bd5-49fd-a943-9c913afe4605-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:59.104467 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.104268    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vfz8\" (UniqueName: \"kubernetes.io/projected/9f34c11f-4bd5-49fd-a943-9c913afe4605-kube-api-access-4vfz8\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:59.779069 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.779031    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv" event={"ID":"9f34c11f-4bd5-49fd-a943-9c913afe4605","Type":"ContainerDied","Data":"b189e5558f49fb473d27d26068987fd7e36357cd1a4427fe415f384f75266781"}
Oct 30 22:34:59.779276 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.779084    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"
Oct 30 22:34:59.779276 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.779107    5060 scope.go:122] "RemoveContainer" containerID="f8c0900034a68850c6e0c96d7c38bf91be222b27aa63e7c002b5c0626d87d7a1"
Oct 30 22:34:59.809991 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.809965    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"]
Oct 30 22:34:59.821206 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:34:59.821182    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-767f9954b4-ff6qv"]
Oct 30 22:35:01.289843 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:01.289804    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f34c11f-4bd5-49fd-a943-9c913afe4605" path="/var/lib/kubelet/pods/9f34c11f-4bd5-49fd-a943-9c913afe4605/volumes"
Oct 30 22:35:04.775213 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:35:04.775168    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:35:07.819638 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:07.819551    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-7489f44cbc-nhvpp_15b5bc12-034a-450c-be74-36e5ec95e170/main/0.log"
Oct 30 22:35:07.820018 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:07.819885    5060 generic.go:356] "Generic (PLEG): container finished" podID="15b5bc12-034a-450c-be74-36e5ec95e170" containerID="e2f69ebc3d11a4f0858f46fcc0eeda3613b95b0181e423107b31e49209f53104" exitCode=137
Oct 30 22:35:07.820018 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:07.819958    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp" event={"ID":"15b5bc12-034a-450c-be74-36e5ec95e170","Type":"ContainerDied","Data":"e2f69ebc3d11a4f0858f46fcc0eeda3613b95b0181e423107b31e49209f53104"}
Oct 30 22:35:08.102235 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.102209    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-7489f44cbc-nhvpp_15b5bc12-034a-450c-be74-36e5ec95e170/main/0.log"
Oct 30 22:35:08.104834 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.104814    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:35:08.184433 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.184395    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-dshm\") pod \"15b5bc12-034a-450c-be74-36e5ec95e170\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") "
Oct 30 22:35:08.184648 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.184468    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-home\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-home\") pod \"15b5bc12-034a-450c-be74-36e5ec95e170\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") "
Oct 30 22:35:08.184648 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.184576    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-model-cache\") pod \"15b5bc12-034a-450c-be74-36e5ec95e170\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") "
Oct 30 22:35:08.184648 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.184628    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-kserve-provision-location\") pod \"15b5bc12-034a-450c-be74-36e5ec95e170\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") "
Oct 30 22:35:08.184823 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.184669    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-tmp-dir\") pod \"15b5bc12-034a-450c-be74-36e5ec95e170\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") "
Oct 30 22:35:08.184823 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.184707    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b5bc12-034a-450c-be74-36e5ec95e170-kube-api-access-9dzfh\" (UniqueName: \"kubernetes.io/projected/15b5bc12-034a-450c-be74-36e5ec95e170-kube-api-access-9dzfh\") pod \"15b5bc12-034a-450c-be74-36e5ec95e170\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") "
Oct 30 22:35:08.184823 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.184747    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b5bc12-034a-450c-be74-36e5ec95e170-tls-certs\" (UniqueName: \"kubernetes.io/secret/15b5bc12-034a-450c-be74-36e5ec95e170-tls-certs\") pod \"15b5bc12-034a-450c-be74-36e5ec95e170\" (UID: \"15b5bc12-034a-450c-be74-36e5ec95e170\") "
Oct 30 22:35:08.184975 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.184823    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-model-cache" pod "15b5bc12-034a-450c-be74-36e5ec95e170" (UID: "15b5bc12-034a-450c-be74-36e5ec95e170"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:08.184975 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.184842    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-home" pod "15b5bc12-034a-450c-be74-36e5ec95e170" (UID: "15b5bc12-034a-450c-be74-36e5ec95e170"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:08.185112 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.185091    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:08.185175 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.185115    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:08.185226 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.185188    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-tmp-dir" pod "15b5bc12-034a-450c-be74-36e5ec95e170" (UID: "15b5bc12-034a-450c-be74-36e5ec95e170"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:08.186823 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.186798    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-dshm" pod "15b5bc12-034a-450c-be74-36e5ec95e170" (UID: "15b5bc12-034a-450c-be74-36e5ec95e170"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:08.186925 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.186896    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b5bc12-034a-450c-be74-36e5ec95e170-tls-certs" pod "15b5bc12-034a-450c-be74-36e5ec95e170" (UID: "15b5bc12-034a-450c-be74-36e5ec95e170"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:08.187119 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.187097    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b5bc12-034a-450c-be74-36e5ec95e170-kube-api-access-9dzfh" pod "15b5bc12-034a-450c-be74-36e5ec95e170" (UID: "15b5bc12-034a-450c-be74-36e5ec95e170"). InnerVolumeSpecName "kube-api-access-9dzfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:08.246575 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.246537    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-kserve-provision-location" pod "15b5bc12-034a-450c-be74-36e5ec95e170" (UID: "15b5bc12-034a-450c-be74-36e5ec95e170"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:08.285491 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.285459    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:08.285491 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.285488    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:08.285491 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.285499    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dzfh\" (UniqueName: \"kubernetes.io/projected/15b5bc12-034a-450c-be74-36e5ec95e170-kube-api-access-9dzfh\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:08.285726 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.285520    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15b5bc12-034a-450c-be74-36e5ec95e170-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:08.285726 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.285528    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15b5bc12-034a-450c-be74-36e5ec95e170-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:08.826213 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.826185    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-7489f44cbc-nhvpp_15b5bc12-034a-450c-be74-36e5ec95e170/main/0.log"
Oct 30 22:35:08.826677 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.826641    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp" event={"ID":"15b5bc12-034a-450c-be74-36e5ec95e170","Type":"ContainerDied","Data":"6919521f4954a443b0966c3614c67721e9da48b04e4d187d3471b8724eb5cd3b"}
Oct 30 22:35:08.826677 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.826659    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"
Oct 30 22:35:08.826797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.826698    5060 scope.go:122] "RemoveContainer" containerID="e2f69ebc3d11a4f0858f46fcc0eeda3613b95b0181e423107b31e49209f53104"
Oct 30 22:35:08.835908 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.835891    5060 scope.go:122] "RemoveContainer" containerID="9af02433661b8744c9488d399c62854799a4f00fcb09c0acb9db591b88f4c631"
Oct 30 22:35:08.853967 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.853939    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"]
Oct 30 22:35:08.858921 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:08.858897    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7489f44cbc-nhvpp"]
Oct 30 22:35:09.289624 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:09.289593    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b5bc12-034a-450c-be74-36e5ec95e170" path="/var/lib/kubelet/pods/15b5bc12-034a-450c-be74-36e5ec95e170/volumes"
Oct 30 22:35:15.415081 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.415047    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"]
Oct 30 22:35:15.415565 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.415454    5060 cpu_manager.go:414] "containerMap: removing container" podUID="15b5bc12-034a-450c-be74-36e5ec95e170" containerName="main"
Oct 30 22:35:15.415565 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.415466    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b5bc12-034a-450c-be74-36e5ec95e170" containerName="main"
Oct 30 22:35:15.415565 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.415484    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9f34c11f-4bd5-49fd-a943-9c913afe4605" containerName="main"
Oct 30 22:35:15.415565 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.415490    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f34c11f-4bd5-49fd-a943-9c913afe4605" containerName="main"
Oct 30 22:35:15.415565 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.415532    5060 cpu_manager.go:414] "containerMap: removing container" podUID="15b5bc12-034a-450c-be74-36e5ec95e170" containerName="storage-initializer"
Oct 30 22:35:15.415565 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.415540    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b5bc12-034a-450c-be74-36e5ec95e170" containerName="storage-initializer"
Oct 30 22:35:15.415883 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.415650    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b5bc12-034a-450c-be74-36e5ec95e170" containerName="main"
Oct 30 22:35:15.415883 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.415659    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f34c11f-4bd5-49fd-a943-9c913afe4605" containerName="main"
Oct 30 22:35:15.424012 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.423987    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.433080 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.433059    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:35:15.433194 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.433176    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Oct 30 22:35:15.434400 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.434380    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Oct 30 22:35:15.439274 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.439255    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"]
Oct 30 22:35:15.439451 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.439434    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"seaweedfs-s3-creds\""
Oct 30 22:35:15.439705 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.439679    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Oct 30 22:35:15.439992 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.439977    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-4t64m\""
Oct 30 22:35:15.548297 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.548255    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.548490 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.548302    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.548490 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.548323    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.548649 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.548478    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.548649 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.548548    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcg6l\" (UniqueName: \"kubernetes.io/projected/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kube-api-access-jcg6l\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.548759 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.548641    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a49727-6a3b-41d3-9ec7-5551cfe7294c-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.548759 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.548679    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.548759 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.548748    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.649839 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.649804    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650003 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.649848    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcg6l\" (UniqueName: \"kubernetes.io/projected/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kube-api-access-jcg6l\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650003 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.649871    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a49727-6a3b-41d3-9ec7-5551cfe7294c-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650003 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.649895    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650003 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.649924    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650003 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.649972    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650245 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.650005    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650245 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.650029    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650346 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.650324    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650401 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.650355    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650478 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.650458    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650554 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.650531    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.650872 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.650843    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a49727-6a3b-41d3-9ec7-5551cfe7294c-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.652579 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.652554    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.652921 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.652900    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.676097 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.676012    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcg6l\" (UniqueName: \"kubernetes.io/projected/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kube-api-access-jcg6l\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:15.737957 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:15.737923    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:16.026477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:16.026433    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"]
Oct 30 22:35:16.031169 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:35:16.031144    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a49727_6a3b_41d3_9ec7_5551cfe7294c.slice/crio-02a7738b6fdfbd8ac43896e8393bd14529a8058e54b1ac4fb9abce98672e64cb WatchSource:0}: Error finding container 02a7738b6fdfbd8ac43896e8393bd14529a8058e54b1ac4fb9abce98672e64cb: Status 404 returned error can't find the container with id 02a7738b6fdfbd8ac43896e8393bd14529a8058e54b1ac4fb9abce98672e64cb
Oct 30 22:35:16.865373 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:16.865335    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" event={"ID":"09a49727-6a3b-41d3-9ec7-5551cfe7294c","Type":"ContainerStarted","Data":"5e97cbc8d23c6fae4b342ee2c8a9147e5b593023c280b677562545e7d059dbea"}
Oct 30 22:35:16.865838 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:16.865382    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" event={"ID":"09a49727-6a3b-41d3-9ec7-5551cfe7294c","Type":"ContainerStarted","Data":"02a7738b6fdfbd8ac43896e8393bd14529a8058e54b1ac4fb9abce98672e64cb"}
Oct 30 22:35:18.876408 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:18.876375    5060 generic.go:356] "Generic (PLEG): container finished" podID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerID="5e97cbc8d23c6fae4b342ee2c8a9147e5b593023c280b677562545e7d059dbea" exitCode=0
Oct 30 22:35:18.876801 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:18.876451    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" event={"ID":"09a49727-6a3b-41d3-9ec7-5551cfe7294c","Type":"ContainerDied","Data":"5e97cbc8d23c6fae4b342ee2c8a9147e5b593023c280b677562545e7d059dbea"}
Oct 30 22:35:19.884388 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:19.884344    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" event={"ID":"09a49727-6a3b-41d3-9ec7-5551cfe7294c","Type":"ContainerStarted","Data":"5a50b3fe94e8339bddc93affa363e228c507011227b963753d23b01d1ccd9b90"}
Oct 30 22:35:19.914501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:19.914430    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" podStartSLOduration=2.635126783 podStartE2EDuration="4.914411127s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.279284344s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:19.912783642 +0000 UTC m=+1000.732521460" watchObservedRunningTime="2026-10-30 22:35:19.914411127 +0000 UTC m=+1000.734148948"
Oct 30 22:35:20.738206 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:20.738174    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:25.738701 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:25.738661    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:35:25.740123 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:25.740097    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Oct 30 22:35:34.775350 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:35:34.775306    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:35:35.738382 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:35.738332    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Oct 30 22:35:45.738922 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:45.738882    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Oct 30 22:35:55.738417 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:35:55.738335    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Oct 30 22:36:04.777312 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:36:04.777269    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:36:05.738411 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:05.738372    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Oct 30 22:36:15.738758 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:15.738711    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Oct 30 22:36:25.738415 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:25.738375    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Oct 30 22:36:34.774922 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:36:34.774889    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:36:35.748314 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:35.748281    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:36:35.755262 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:35.755231    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:36:39.537305 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:39.537272    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"]
Oct 30 22:36:54.556988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:54.556948    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main" containerID="cri-o://5a50b3fe94e8339bddc93affa363e228c507011227b963753d23b01d1ccd9b90" gracePeriod=45
Oct 30 22:36:54.749062 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:54.749021    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Oct 30 22:36:55.345003 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.344966    5060 generic.go:356] "Generic (PLEG): container finished" podID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerID="5a50b3fe94e8339bddc93affa363e228c507011227b963753d23b01d1ccd9b90" exitCode=0
Oct 30 22:36:55.345155 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.345014    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" event={"ID":"09a49727-6a3b-41d3-9ec7-5551cfe7294c","Type":"ContainerDied","Data":"5a50b3fe94e8339bddc93affa363e228c507011227b963753d23b01d1ccd9b90"}
Oct 30 22:36:55.404584 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.404557    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:36:55.560375 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560347    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tmp-dir\") pod \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") "
Oct 30 22:36:55.560809 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560384    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a49727-6a3b-41d3-9ec7-5551cfe7294c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a49727-6a3b-41d3-9ec7-5551cfe7294c-cabundle-cert\") pod \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") "
Oct 30 22:36:55.560809 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560439    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-model-cache\") pod \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") "
Oct 30 22:36:55.560809 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560476    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kserve-provision-location\") pod \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") "
Oct 30 22:36:55.560809 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560532    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-dshm\") pod \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") "
Oct 30 22:36:55.560809 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560566    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tls-certs\" (UniqueName: \"kubernetes.io/secret/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tls-certs\") pod \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") "
Oct 30 22:36:55.560809 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560590    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kube-api-access-jcg6l\" (UniqueName: \"kubernetes.io/projected/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kube-api-access-jcg6l\") pod \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") "
Oct 30 22:36:55.560809 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560617    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-home\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-home\") pod \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\" (UID: \"09a49727-6a3b-41d3-9ec7-5551cfe7294c\") "
Oct 30 22:36:55.560809 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560743    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-model-cache" pod "09a49727-6a3b-41d3-9ec7-5551cfe7294c" (UID: "09a49727-6a3b-41d3-9ec7-5551cfe7294c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:55.561224 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560825    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a49727-6a3b-41d3-9ec7-5551cfe7294c-cabundle-cert" pod "09a49727-6a3b-41d3-9ec7-5551cfe7294c" (UID: "09a49727-6a3b-41d3-9ec7-5551cfe7294c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:55.561224 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560916    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a49727-6a3b-41d3-9ec7-5551cfe7294c-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:55.561224 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.560970    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:55.561224 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.561155    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-home" pod "09a49727-6a3b-41d3-9ec7-5551cfe7294c" (UID: "09a49727-6a3b-41d3-9ec7-5551cfe7294c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:55.562827 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.562796    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-dshm" pod "09a49727-6a3b-41d3-9ec7-5551cfe7294c" (UID: "09a49727-6a3b-41d3-9ec7-5551cfe7294c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:55.563150 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.563128    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kube-api-access-jcg6l" pod "09a49727-6a3b-41d3-9ec7-5551cfe7294c" (UID: "09a49727-6a3b-41d3-9ec7-5551cfe7294c"). InnerVolumeSpecName "kube-api-access-jcg6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:55.563277 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.563257    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tls-certs" pod "09a49727-6a3b-41d3-9ec7-5551cfe7294c" (UID: "09a49727-6a3b-41d3-9ec7-5551cfe7294c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:55.570670 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.570649    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kserve-provision-location" pod "09a49727-6a3b-41d3-9ec7-5551cfe7294c" (UID: "09a49727-6a3b-41d3-9ec7-5551cfe7294c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:55.576797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.576773    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tmp-dir" pod "09a49727-6a3b-41d3-9ec7-5551cfe7294c" (UID: "09a49727-6a3b-41d3-9ec7-5551cfe7294c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:55.661872 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.661845    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:55.661872 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.661871    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:55.662033 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.661885    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:55.662033 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.661897    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a49727-6a3b-41d3-9ec7-5551cfe7294c-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:55.662033 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.661908    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcg6l\" (UniqueName: \"kubernetes.io/projected/09a49727-6a3b-41d3-9ec7-5551cfe7294c-kube-api-access-jcg6l\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:55.662033 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:55.661919    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a49727-6a3b-41d3-9ec7-5551cfe7294c-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:56.351325 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:56.351294    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"
Oct 30 22:36:56.351486 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:56.351294    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2" event={"ID":"09a49727-6a3b-41d3-9ec7-5551cfe7294c","Type":"ContainerDied","Data":"02a7738b6fdfbd8ac43896e8393bd14529a8058e54b1ac4fb9abce98672e64cb"}
Oct 30 22:36:56.351486 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:56.351422    5060 scope.go:122] "RemoveContainer" containerID="5a50b3fe94e8339bddc93affa363e228c507011227b963753d23b01d1ccd9b90"
Oct 30 22:36:56.361074 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:56.361057    5060 scope.go:122] "RemoveContainer" containerID="5e97cbc8d23c6fae4b342ee2c8a9147e5b593023c280b677562545e7d059dbea"
Oct 30 22:36:56.378382 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:56.378356    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"]
Oct 30 22:36:56.387537 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:56.387499    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-66b55b88f9m5tq2"]
Oct 30 22:36:57.289142 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:36:57.289102    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" path="/var/lib/kubelet/pods/09a49727-6a3b-41d3-9ec7-5551cfe7294c/volumes"
Oct 30 22:37:04.774493 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:37:04.774458    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:37:32.704561 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.704528    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"]
Oct 30 22:37:32.704913 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.704898    5060 cpu_manager.go:414] "containerMap: removing container" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="storage-initializer"
Oct 30 22:37:32.704953 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.704911    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="storage-initializer"
Oct 30 22:37:32.704953 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.704936    5060 cpu_manager.go:414] "containerMap: removing container" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main"
Oct 30 22:37:32.704953 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.704941    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main"
Oct 30 22:37:32.705046 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.705012    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a49727-6a3b-41d3-9ec7-5551cfe7294c" containerName="main"
Oct 30 22:37:32.707672 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.707652    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.724439 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.724416    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Oct 30 22:37:32.724564 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.724441    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-xcq9r\""
Oct 30 22:37:32.724564 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.724525    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Oct 30 22:37:32.724659 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.724646    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Oct 30 22:37:32.735806 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.735777    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"]
Oct 30 22:37:32.742989 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.742971    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.743078 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.743004    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3717fb14-a1f7-49cc-befb-694ffd830d4d-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.743078 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.743020    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdbs2\" (UniqueName: \"kubernetes.io/projected/3717fb14-a1f7-49cc-befb-694ffd830d4d-kube-api-access-mdbs2\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.743078 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.743058    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.743199 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.743156    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.743234 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.743211    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.843905 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.843859    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.844127 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.843926    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.844127 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.843956    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.844127 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.843994    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3717fb14-a1f7-49cc-befb-694ffd830d4d-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.844127 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.844079    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdbs2\" (UniqueName: \"kubernetes.io/projected/3717fb14-a1f7-49cc-befb-694ffd830d4d-kube-api-access-mdbs2\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.844344 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.844128    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.844344 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.844334    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.844454 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.844374    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.844557 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.844535    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.846718 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.846701    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.847082 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.847065    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3717fb14-a1f7-49cc-befb-694ffd830d4d-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:32.857782 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:32.857759    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdbs2\" (UniqueName: \"kubernetes.io/projected/3717fb14-a1f7-49cc-befb-694ffd830d4d-kube-api-access-mdbs2\") pod \"fc-smoke-test-kserve-7d8f9d9664-l7klz\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:33.020133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:33.020047    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:33.302673 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:33.302652    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"]
Oct 30 22:37:33.304928 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:37:33.304902    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3717fb14_a1f7_49cc_befb_694ffd830d4d.slice/crio-a6a91cc9c9d64c57b31122bb8dfe076619a77b5fa9609096149fcdd84ee21609 WatchSource:0}: Error finding container a6a91cc9c9d64c57b31122bb8dfe076619a77b5fa9609096149fcdd84ee21609: Status 404 returned error can't find the container with id a6a91cc9c9d64c57b31122bb8dfe076619a77b5fa9609096149fcdd84ee21609
Oct 30 22:37:33.513108 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:33.513071    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz" event={"ID":"3717fb14-a1f7-49cc-befb-694ffd830d4d","Type":"ContainerStarted","Data":"c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893"}
Oct 30 22:37:33.513267 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:33.513147    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz" event={"ID":"3717fb14-a1f7-49cc-befb-694ffd830d4d","Type":"ContainerStarted","Data":"a6a91cc9c9d64c57b31122bb8dfe076619a77b5fa9609096149fcdd84ee21609"}
Oct 30 22:37:33.539735 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:33.539692    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz" podStartSLOduration=1.5396787029999999 podStartE2EDuration="1.539678703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:33.539120238 +0000 UTC m=+1134.358858057" watchObservedRunningTime="2026-10-30 22:37:33.539678703 +0000 UTC m=+1134.359416534"
Oct 30 22:37:34.021532 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:34.020611    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:34.832078 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:37:34.832035    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:37:43.020252 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:43.020215    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:43.032252 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:43.032228    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:37:43.572423 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:37:43.572390    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:38:04.774792 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:38:04.774760    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:38:13.592427 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:13.592393    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"]
Oct 30 22:38:13.601281 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:38:13.601245    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893" cmd=["/bin/sleep","15"]
Oct 30 22:38:13.601411 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:13.601292    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:38:13.601411 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:38:13.601305    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz" podUID="3717fb14-a1f7-49cc-befb-694ffd830d4d" containerName="main" containerID="cri-o://c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893"
Oct 30 22:38:13.601411 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:13.601345    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz" podUID="3717fb14-a1f7-49cc-befb-694ffd830d4d" containerName="main" containerID="cri-o://c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893" gracePeriod=60
Oct 30 22:38:14.340546 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.340518    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:38:14.464889 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.464811    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-tmp-dir\") pod \"3717fb14-a1f7-49cc-befb-694ffd830d4d\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") "
Oct 30 22:38:14.464889 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.464859    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-home\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-home\") pod \"3717fb14-a1f7-49cc-befb-694ffd830d4d\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") "
Oct 30 22:38:14.465098 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.464931    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3717fb14-a1f7-49cc-befb-694ffd830d4d-kube-api-access-mdbs2\" (UniqueName: \"kubernetes.io/projected/3717fb14-a1f7-49cc-befb-694ffd830d4d-kube-api-access-mdbs2\") pod \"3717fb14-a1f7-49cc-befb-694ffd830d4d\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") "
Oct 30 22:38:14.465098 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.464981    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3717fb14-a1f7-49cc-befb-694ffd830d4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/3717fb14-a1f7-49cc-befb-694ffd830d4d-tls-certs\") pod \"3717fb14-a1f7-49cc-befb-694ffd830d4d\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") "
Oct 30 22:38:14.465098 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.465038    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-model-cache\") pod \"3717fb14-a1f7-49cc-befb-694ffd830d4d\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") "
Oct 30 22:38:14.465098 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.465082    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-dshm\") pod \"3717fb14-a1f7-49cc-befb-694ffd830d4d\" (UID: \"3717fb14-a1f7-49cc-befb-694ffd830d4d\") "
Oct 30 22:38:14.465350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.465057    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-tmp-dir" pod "3717fb14-a1f7-49cc-befb-694ffd830d4d" (UID: "3717fb14-a1f7-49cc-befb-694ffd830d4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:14.465350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.465238    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-model-cache" pod "3717fb14-a1f7-49cc-befb-694ffd830d4d" (UID: "3717fb14-a1f7-49cc-befb-694ffd830d4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:14.465350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.465139    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-home" pod "3717fb14-a1f7-49cc-befb-694ffd830d4d" (UID: "3717fb14-a1f7-49cc-befb-694ffd830d4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:14.465494 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.465414    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:14.465494 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.465433    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:14.465494 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.465447    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:14.467146 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.467125    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-dshm" pod "3717fb14-a1f7-49cc-befb-694ffd830d4d" (UID: "3717fb14-a1f7-49cc-befb-694ffd830d4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:14.467146 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.467139    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3717fb14-a1f7-49cc-befb-694ffd830d4d-tls-certs" pod "3717fb14-a1f7-49cc-befb-694ffd830d4d" (UID: "3717fb14-a1f7-49cc-befb-694ffd830d4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:14.467266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.467152    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3717fb14-a1f7-49cc-befb-694ffd830d4d-kube-api-access-mdbs2" pod "3717fb14-a1f7-49cc-befb-694ffd830d4d" (UID: "3717fb14-a1f7-49cc-befb-694ffd830d4d"). InnerVolumeSpecName "kube-api-access-mdbs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:14.565865 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.565840    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdbs2\" (UniqueName: \"kubernetes.io/projected/3717fb14-a1f7-49cc-befb-694ffd830d4d-kube-api-access-mdbs2\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:14.565865 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.565863    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3717fb14-a1f7-49cc-befb-694ffd830d4d-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:14.566020 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.565872    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3717fb14-a1f7-49cc-befb-694ffd830d4d-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:14.703000 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.702965    5060 generic.go:356] "Generic (PLEG): container finished" podID="3717fb14-a1f7-49cc-befb-694ffd830d4d" containerID="c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893" exitCode=0
Oct 30 22:38:14.703435 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.703008    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"
Oct 30 22:38:14.703435 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.703037    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz" event={"ID":"3717fb14-a1f7-49cc-befb-694ffd830d4d","Type":"ContainerDied","Data":"c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893"}
Oct 30 22:38:14.703435 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.703090    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz" event={"ID":"3717fb14-a1f7-49cc-befb-694ffd830d4d","Type":"ContainerDied","Data":"a6a91cc9c9d64c57b31122bb8dfe076619a77b5fa9609096149fcdd84ee21609"}
Oct 30 22:38:14.703435 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.703123    5060 scope.go:122] "RemoveContainer" containerID="c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893"
Oct 30 22:38:14.711845 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.711826    5060 scope.go:122] "RemoveContainer" containerID="c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893"
Oct 30 22:38:14.712066 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:38:14.712048    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893\": container with ID starting with c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893 not found: ID does not exist" containerID="c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893"
Oct 30 22:38:14.712125 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.712077    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893"} err="failed to get container status \"c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893\": rpc error: code = NotFound desc = could not find container \"c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893\": container with ID starting with c6a1601d651b61bc3da73dddf570a5f6b717b3e89a94811c38689ab6b16a0893 not found: ID does not exist"
Oct 30 22:38:14.726628 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.726573    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"]
Oct 30 22:38:14.728625 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:14.728605    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-l7klz"]
Oct 30 22:38:15.289243 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:15.289211    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3717fb14-a1f7-49cc-befb-694ffd830d4d" path="/var/lib/kubelet/pods/3717fb14-a1f7-49cc-befb-694ffd830d4d/volumes"
Oct 30 22:38:34.775490 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:38:34.775452    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:38:39.394238 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:39.394215    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:38:39.396248 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:38:39.396228    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:39:04.775571 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:39:04.775463    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:39:07.604372 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.604335    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"]
Oct 30 22:39:07.604749 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.604711    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3717fb14-a1f7-49cc-befb-694ffd830d4d" containerName="main"
Oct 30 22:39:07.604749 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.604724    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3717fb14-a1f7-49cc-befb-694ffd830d4d" containerName="main"
Oct 30 22:39:07.604822 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.604789    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3717fb14-a1f7-49cc-befb-694ffd830d4d" containerName="main"
Oct 30 22:39:07.607753 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.607736    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.613252 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.613222    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Oct 30 22:39:07.616202 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.616182    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Oct 30 22:39:07.617627 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.617609    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Oct 30 22:39:07.620691 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.620670    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"]
Oct 30 22:39:07.665672 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.665604    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-44642\""
Oct 30 22:39:07.683414 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.683387    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-dshm\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.683545 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.683421    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-model-cache\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.683545 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.683449    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-home\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.683623 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.683595    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5tl\" (UniqueName: \"kubernetes.io/projected/d77bd2f7-ddd0-47df-802d-98a5907c6500-kube-api-access-cr5tl\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.683656 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.683631    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-tmp-dir\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.683656 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.683651    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d77bd2f7-ddd0-47df-802d-98a5907c6500-tls-certs\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.784656 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.784623    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5tl\" (UniqueName: \"kubernetes.io/projected/d77bd2f7-ddd0-47df-802d-98a5907c6500-kube-api-access-cr5tl\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.784818 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.784665    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-tmp-dir\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.784818 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.784684    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d77bd2f7-ddd0-47df-802d-98a5907c6500-tls-certs\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.784818 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.784765    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-dshm\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.784818 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.784794    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-model-cache\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.784972 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.784829    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-home\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.785123 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.785093    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-tmp-dir\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.785246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.785151    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-model-cache\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.785246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.785172    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-home\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.786982 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.786958    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-dshm\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.787263 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.787244    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d77bd2f7-ddd0-47df-802d-98a5907c6500-tls-certs\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.797976 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.797948    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5tl\" (UniqueName: \"kubernetes.io/projected/d77bd2f7-ddd0-47df-802d-98a5907c6500-kube-api-access-cr5tl\") pod \"fc-concurrency-test-kserve-5485c7876d-lh56t\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:07.920914 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:07.920832    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:08.200112 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:08.200088    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"]
Oct 30 22:39:08.202593 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:39:08.202560    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77bd2f7_ddd0_47df_802d_98a5907c6500.slice/crio-a82c0394f7b5337ab8720090629484a8229cfa9492bd573f303bca5bcdf1274f WatchSource:0}: Error finding container a82c0394f7b5337ab8720090629484a8229cfa9492bd573f303bca5bcdf1274f: Status 404 returned error can't find the container with id a82c0394f7b5337ab8720090629484a8229cfa9492bd573f303bca5bcdf1274f
Oct 30 22:39:08.940568 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:08.940532    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t" event={"ID":"d77bd2f7-ddd0-47df-802d-98a5907c6500","Type":"ContainerStarted","Data":"af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b"}
Oct 30 22:39:08.940957 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:08.940584    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t" event={"ID":"d77bd2f7-ddd0-47df-802d-98a5907c6500","Type":"ContainerStarted","Data":"a82c0394f7b5337ab8720090629484a8229cfa9492bd573f303bca5bcdf1274f"}
Oct 30 22:39:08.972532 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:08.971637    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t" podStartSLOduration=1.971616743 podStartE2EDuration="1.971616743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:08.960531801 +0000 UTC m=+1229.780269644" watchObservedRunningTime="2026-10-30 22:39:08.971616743 +0000 UTC m=+1229.791354567"
Oct 30 22:39:09.921723 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:09.921673    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:17.921735 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:17.921700    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:17.933880 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:17.933857    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:18.069305 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:18.069276    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:29.839772 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.839732    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-mxxhw-rrcns"]
Oct 30 22:39:29.843011 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.842994    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.848442 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.848423    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dhvzl\""
Oct 30 22:39:29.853920 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.853899    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-mxxhw-rrcns"]
Oct 30 22:39:29.879328 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.879301    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-service-ca-bundle\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.879420 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.879346    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-trusted-ca-bundle\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.879420 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.879369    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf46t\" (UniqueName: \"kubernetes.io/projected/6b5b9230-9003-4591-8440-01f8a2ccf26e-kube-api-access-xf46t\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.879549 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.879527    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6b5b9230-9003-4591-8440-01f8a2ccf26e-archives-path\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.980624 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.980598    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6b5b9230-9003-4591-8440-01f8a2ccf26e-archives-path\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.980740 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.980637    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-service-ca-bundle\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.980740 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.980670    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-trusted-ca-bundle\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.980740 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.980692    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf46t\" (UniqueName: \"kubernetes.io/projected/6b5b9230-9003-4591-8440-01f8a2ccf26e-kube-api-access-xf46t\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.981042 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.981022    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6b5b9230-9003-4591-8440-01f8a2ccf26e-archives-path\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.981312 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.981293    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-service-ca-bundle\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.981367 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.981351    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-trusted-ca-bundle\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:29.994009 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:29.993985    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf46t\" (UniqueName: \"kubernetes.io/projected/6b5b9230-9003-4591-8440-01f8a2ccf26e-kube-api-access-xf46t\") pod \"periodic-gathering-mxxhw-rrcns\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") " pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:30.154944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:30.154860    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:39:30.433859 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:30.433837    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-mxxhw-rrcns"]
Oct 30 22:39:30.436446 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:39:30.436415    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5b9230_9003_4591_8440_01f8a2ccf26e.slice/crio-985e2941dd43295049b8fa7f3568350a9e12bda39478591abe02a7a34a01240e WatchSource:0}: Error finding container 985e2941dd43295049b8fa7f3568350a9e12bda39478591abe02a7a34a01240e: Status 404 returned error can't find the container with id 985e2941dd43295049b8fa7f3568350a9e12bda39478591abe02a7a34a01240e
Oct 30 22:39:31.115859 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:31.115824    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mxxhw-rrcns" event={"ID":"6b5b9230-9003-4591-8440-01f8a2ccf26e","Type":"ContainerStarted","Data":"93442c23a498598e2c36544ad95a7a0a101ac8ade99c97a1eaaf19cb0c7857a9"}
Oct 30 22:39:31.116249 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:31.115877    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mxxhw-rrcns" event={"ID":"6b5b9230-9003-4591-8440-01f8a2ccf26e","Type":"ContainerStarted","Data":"985e2941dd43295049b8fa7f3568350a9e12bda39478591abe02a7a34a01240e"}
Oct 30 22:39:31.135244 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:31.135197    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-mxxhw-rrcns" podStartSLOduration=2.135184267 podStartE2EDuration="2.135184267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:31.134569409 +0000 UTC m=+1251.954307227" watchObservedRunningTime="2026-10-30 22:39:31.135184267 +0000 UTC m=+1251.954922064"
Oct 30 22:39:33.743168 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.743133    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"]
Oct 30 22:39:33.746927 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.746898    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:39:33.760036 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.760006    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"]
Oct 30 22:39:33.813984 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.813950    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl9d5\" (UniqueName: \"kubernetes.io/projected/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-kube-api-access-wl9d5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c\" (UID: \"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:39:33.814122 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.814015    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c\" (UID: \"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:39:33.844674 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.844651    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Oct 30 22:39:33.844793 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.844739    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Oct 30 22:39:33.844793 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.844759    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Oct 30 22:39:33.844890 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.844843    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-5lf77\""
Oct 30 22:39:33.844890 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.844882    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hb45s\""
Oct 30 22:39:33.915447 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.915411    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c\" (UID: \"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:39:33.915607 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.915559    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl9d5\" (UniqueName: \"kubernetes.io/projected/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-kube-api-access-wl9d5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c\" (UID: \"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:39:33.918718 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.918690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c\" (UID: \"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:39:33.940043 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:33.940015    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl9d5\" (UniqueName: \"kubernetes.io/projected/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-kube-api-access-wl9d5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c\" (UID: \"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:39:34.060733 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:34.060598    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:39:34.342520 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:34.342482    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"]
Oct 30 22:39:34.345122 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:39:34.345093    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556fd1fc_a30b_42a7_a21f_59e3ef6f7a54.slice/crio-540ce6dd5bf73344180699766104feb35e504a4e79f0f465f00d88f1eda3cd29 WatchSource:0}: Error finding container 540ce6dd5bf73344180699766104feb35e504a4e79f0f465f00d88f1eda3cd29: Status 404 returned error can't find the container with id 540ce6dd5bf73344180699766104feb35e504a4e79f0f465f00d88f1eda3cd29
Oct 30 22:39:34.347641 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:34.347622    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:39:34.774972 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:39:34.774938    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:39:35.139388 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:35.139298    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" event={"ID":"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54","Type":"ContainerStarted","Data":"540ce6dd5bf73344180699766104feb35e504a4e79f0f465f00d88f1eda3cd29"}
Oct 30 22:39:36.145442 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:36.145368    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" event={"ID":"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54","Type":"ContainerStarted","Data":"86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929"}
Oct 30 22:39:36.145906 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:36.145502    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:39:36.165795 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:36.165583    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" podStartSLOduration=1.7643312450000002 podStartE2EDuration="3.165572113s" totalImagesPullingTime="1.401240868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:36.164948487 +0000 UTC m=+1256.984686306" watchObservedRunningTime="2026-10-30 22:39:36.165572113 +0000 UTC m=+1256.985309913"
Oct 30 22:39:48.311912 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.311873    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"]
Oct 30 22:39:48.321023 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:39:48.320988    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b" cmd=["/bin/sleep","15"]
Oct 30 22:39:48.321152 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.321037    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:39:48.321152 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:39:48.321049    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t" podUID="d77bd2f7-ddd0-47df-802d-98a5907c6500" containerName="main" containerID="cri-o://af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b"
Oct 30 22:39:48.321152 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.321105    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t" podUID="d77bd2f7-ddd0-47df-802d-98a5907c6500" containerName="main" containerID="cri-o://af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b" gracePeriod=60
Oct 30 22:39:48.665440 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.665417    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:48.763029 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.762985    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-tmp-dir\") pod \"d77bd2f7-ddd0-47df-802d-98a5907c6500\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") "
Oct 30 22:39:48.763227 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.763119    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-home\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-home\") pod \"d77bd2f7-ddd0-47df-802d-98a5907c6500\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") "
Oct 30 22:39:48.763227 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.763171    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d77bd2f7-ddd0-47df-802d-98a5907c6500-tls-certs\" (UniqueName: \"kubernetes.io/secret/d77bd2f7-ddd0-47df-802d-98a5907c6500-tls-certs\") pod \"d77bd2f7-ddd0-47df-802d-98a5907c6500\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") "
Oct 30 22:39:48.763227 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.763215    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-dshm\") pod \"d77bd2f7-ddd0-47df-802d-98a5907c6500\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") "
Oct 30 22:39:48.763407 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.763274    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-tmp-dir" pod "d77bd2f7-ddd0-47df-802d-98a5907c6500" (UID: "d77bd2f7-ddd0-47df-802d-98a5907c6500"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:48.763407 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.763294    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-model-cache\") pod \"d77bd2f7-ddd0-47df-802d-98a5907c6500\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") "
Oct 30 22:39:48.763407 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.763380    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77bd2f7-ddd0-47df-802d-98a5907c6500-kube-api-access-cr5tl\" (UniqueName: \"kubernetes.io/projected/d77bd2f7-ddd0-47df-802d-98a5907c6500-kube-api-access-cr5tl\") pod \"d77bd2f7-ddd0-47df-802d-98a5907c6500\" (UID: \"d77bd2f7-ddd0-47df-802d-98a5907c6500\") "
Oct 30 22:39:48.763587 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.763562    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-model-cache" pod "d77bd2f7-ddd0-47df-802d-98a5907c6500" (UID: "d77bd2f7-ddd0-47df-802d-98a5907c6500"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:48.763770 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.763717    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-home" pod "d77bd2f7-ddd0-47df-802d-98a5907c6500" (UID: "d77bd2f7-ddd0-47df-802d-98a5907c6500"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:48.763770 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.763725    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.763770 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.763751    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.766057 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.766028    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77bd2f7-ddd0-47df-802d-98a5907c6500-tls-certs" pod "d77bd2f7-ddd0-47df-802d-98a5907c6500" (UID: "d77bd2f7-ddd0-47df-802d-98a5907c6500"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:48.766173 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.766071    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-dshm" pod "d77bd2f7-ddd0-47df-802d-98a5907c6500" (UID: "d77bd2f7-ddd0-47df-802d-98a5907c6500"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:48.767132 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.767103    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77bd2f7-ddd0-47df-802d-98a5907c6500-kube-api-access-cr5tl" pod "d77bd2f7-ddd0-47df-802d-98a5907c6500" (UID: "d77bd2f7-ddd0-47df-802d-98a5907c6500"). InnerVolumeSpecName "kube-api-access-cr5tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:48.864368 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.864276    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.864368 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.864307    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d77bd2f7-ddd0-47df-802d-98a5907c6500-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.864368 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.864317    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d77bd2f7-ddd0-47df-802d-98a5907c6500-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:48.864368 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:48.864329    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr5tl\" (UniqueName: \"kubernetes.io/projected/d77bd2f7-ddd0-47df-802d-98a5907c6500-kube-api-access-cr5tl\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:49.208422 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:49.208387    5060 generic.go:356] "Generic (PLEG): container finished" podID="d77bd2f7-ddd0-47df-802d-98a5907c6500" containerID="af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b" exitCode=0
Oct 30 22:39:49.208623 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:49.208436    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"
Oct 30 22:39:49.208623 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:49.208470    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t" event={"ID":"d77bd2f7-ddd0-47df-802d-98a5907c6500","Type":"ContainerDied","Data":"af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b"}
Oct 30 22:39:49.208623 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:49.208534    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t" event={"ID":"d77bd2f7-ddd0-47df-802d-98a5907c6500","Type":"ContainerDied","Data":"a82c0394f7b5337ab8720090629484a8229cfa9492bd573f303bca5bcdf1274f"}
Oct 30 22:39:49.208623 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:49.208556    5060 scope.go:122] "RemoveContainer" containerID="af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b"
Oct 30 22:39:49.217373 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:49.217357    5060 scope.go:122] "RemoveContainer" containerID="af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b"
Oct 30 22:39:49.217621 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:39:49.217603    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b\": container with ID starting with af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b not found: ID does not exist" containerID="af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b"
Oct 30 22:39:49.217690 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:49.217634    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b"} err="failed to get container status \"af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b\": rpc error: code = NotFound desc = could not find container \"af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b\": container with ID starting with af376439ce89ca20400deea45fb921ade0fe93aa2bd8cedc1f20191586013b3b not found: ID does not exist"
Oct 30 22:39:49.231354 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:49.231334    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"]
Oct 30 22:39:49.236428 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:49.236407    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-lh56t"]
Oct 30 22:39:49.295078 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:39:49.295042    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77bd2f7-ddd0-47df-802d-98a5907c6500" path="/var/lib/kubelet/pods/d77bd2f7-ddd0-47df-802d-98a5907c6500/volumes"
Oct 30 22:40:04.775811 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:40:04.775773    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:40:07.151066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:07.151032    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:07.838851 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:07.838818    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"]
Oct 30 22:40:07.840474 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:07.840444    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:17.841296 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:17.841253    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:22.862117 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:22.862030    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerName="main" containerID="cri-o://86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929" gracePeriod=45
Oct 30 22:40:27.841434 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:27.841398    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:34.775933 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:40:34.775890    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:40:37.840835 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:37.840799    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:47.841299 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:47.841262    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:40:53.105854 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.105831    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:40:53.212198 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.212169    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-kube-api-access-wl9d5\" (UniqueName: \"kubernetes.io/projected/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-kube-api-access-wl9d5\") pod \"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54\" (UID: \"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54\") "
Oct 30 22:40:53.212401 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.212257    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-tls-certs\" (UniqueName: \"kubernetes.io/secret/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-tls-certs\") pod \"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54\" (UID: \"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54\") "
Oct 30 22:40:53.214378 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.214353    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-kube-api-access-wl9d5" pod "556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" (UID: "556fd1fc-a30b-42a7-a21f-59e3ef6f7a54"). InnerVolumeSpecName "kube-api-access-wl9d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:53.214482 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.214377    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-tls-certs" pod "556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" (UID: "556fd1fc-a30b-42a7-a21f-59e3ef6f7a54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:53.313687 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.313658    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:53.313687 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.313683    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl9d5\" (UniqueName: \"kubernetes.io/projected/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54-kube-api-access-wl9d5\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:53.491516 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.491419    5060 generic.go:356] "Generic (PLEG): container finished" podID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerID="86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929" exitCode=0
Oct 30 22:40:53.491516 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.491473    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"
Oct 30 22:40:53.491706 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.491528    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" event={"ID":"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54","Type":"ContainerDied","Data":"86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929"}
Oct 30 22:40:53.491706 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.491587    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c" event={"ID":"556fd1fc-a30b-42a7-a21f-59e3ef6f7a54","Type":"ContainerDied","Data":"540ce6dd5bf73344180699766104feb35e504a4e79f0f465f00d88f1eda3cd29"}
Oct 30 22:40:53.491706 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.491608    5060 scope.go:122] "RemoveContainer" containerID="86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929"
Oct 30 22:40:53.500138 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.500121    5060 scope.go:122] "RemoveContainer" containerID="86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929"
Oct 30 22:40:53.500439 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:40:53.500411    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929\": container with ID starting with 86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929 not found: ID does not exist" containerID="86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929"
Oct 30 22:40:53.500523 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.500451    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929"} err="failed to get container status \"86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929\": rpc error: code = NotFound desc = could not find container \"86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929\": container with ID starting with 86dbbdfa37386e2c7d1c790483d81c831e728261ececcf51927f189ddfd1f929 not found: ID does not exist"
Oct 30 22:40:53.510934 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.510911    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"]
Oct 30 22:40:53.515916 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:53.515893    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b62zs4c"]
Oct 30 22:40:55.290041 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:55.290008    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" path="/var/lib/kubelet/pods/556fd1fc-a30b-42a7-a21f-59e3ef6f7a54/volumes"
Oct 30 22:40:59.886708 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.886666    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"]
Oct 30 22:40:59.887297 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.887277    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d77bd2f7-ddd0-47df-802d-98a5907c6500" containerName="main"
Oct 30 22:40:59.887344 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.887300    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77bd2f7-ddd0-47df-802d-98a5907c6500" containerName="main"
Oct 30 22:40:59.887344 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.887313    5060 cpu_manager.go:414] "containerMap: removing container" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerName="main"
Oct 30 22:40:59.887344 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.887320    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerName="main"
Oct 30 22:40:59.887438 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.887410    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77bd2f7-ddd0-47df-802d-98a5907c6500" containerName="main"
Oct 30 22:40:59.887438 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.887423    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="556fd1fc-a30b-42a7-a21f-59e3ef6f7a54" containerName="main"
Oct 30 22:40:59.890745 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.890723    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:40:59.895662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.895635    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Oct 30 22:40:59.895811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.895721    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Oct 30 22:40:59.896996 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.896976    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-wtn5p\""
Oct 30 22:40:59.897131 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.897111    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Oct 30 22:40:59.912397 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.912373    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"]
Oct 30 22:40:59.970980 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.970948    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:40:59.970980 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.970980    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:40:59.971176 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.971003    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9z5\" (UniqueName: \"kubernetes.io/projected/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-kube-api-access-rq9z5\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:40:59.971176 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.971127    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:40:59.971176 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.971171    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:40:59.971278 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:40:59.971250    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.072470 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.072430    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.072666 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.072481    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.072666 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.072546    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.072666 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.072564    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.072666 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.072580    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9z5\" (UniqueName: \"kubernetes.io/projected/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-kube-api-access-rq9z5\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.072666 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.072609    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.073040 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.073020    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-home\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.073107 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.073085    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tmp-dir\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.073150 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.073096    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-model-cache\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.074881 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.074863    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-dshm\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.075155 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.075120    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tls-certs\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.085347 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.085320    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9z5\" (UniqueName: \"kubernetes.io/projected/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-kube-api-access-rq9z5\") pod \"scheduler-custom-template-test-kserve-6667fc56b8-znp6t\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.204840 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.204815    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:00.486431 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.486408    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"]
Oct 30 22:41:00.488771 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:41:00.488743    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e56ebc_d0c0_4d96_9e3b_665f5de81bf4.slice/crio-b113106c4424ee27c40b025a0e19813e7e9831f56ff9b5ebdc961df4627abb44 WatchSource:0}: Error finding container b113106c4424ee27c40b025a0e19813e7e9831f56ff9b5ebdc961df4627abb44: Status 404 returned error can't find the container with id b113106c4424ee27c40b025a0e19813e7e9831f56ff9b5ebdc961df4627abb44
Oct 30 22:41:00.525773 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:00.525744    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t" event={"ID":"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4","Type":"ContainerStarted","Data":"b113106c4424ee27c40b025a0e19813e7e9831f56ff9b5ebdc961df4627abb44"}
Oct 30 22:41:01.531904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:01.531872    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t" event={"ID":"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4","Type":"ContainerStarted","Data":"e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024"}
Oct 30 22:41:02.207955 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:02.207672    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:02.537728 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:02.537644    5060 generic.go:356] "Generic (PLEG): container finished" podID="6b5b9230-9003-4591-8440-01f8a2ccf26e" containerID="93442c23a498598e2c36544ad95a7a0a101ac8ade99c97a1eaaf19cb0c7857a9" exitCode=0
Oct 30 22:41:02.537728 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:02.537717    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mxxhw-rrcns" event={"ID":"6b5b9230-9003-4591-8440-01f8a2ccf26e","Type":"ContainerDied","Data":"93442c23a498598e2c36544ad95a7a0a101ac8ade99c97a1eaaf19cb0c7857a9"}
Oct 30 22:41:02.566526 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:02.566362    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t" podStartSLOduration=3.5663324640000003 podStartE2EDuration="3.566332464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:01.561308318 +0000 UTC m=+1342.381046138" watchObservedRunningTime="2026-10-30 22:41:02.566332464 +0000 UTC m=+1343.386070285"
Oct 30 22:41:03.944961 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:03.944934    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:41:04.037440 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.037406    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-service-ca-bundle\") pod \"6b5b9230-9003-4591-8440-01f8a2ccf26e\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") "
Oct 30 22:41:04.037626 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.037468    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5b9230-9003-4591-8440-01f8a2ccf26e-kube-api-access-xf46t\" (UniqueName: \"kubernetes.io/projected/6b5b9230-9003-4591-8440-01f8a2ccf26e-kube-api-access-xf46t\") pod \"6b5b9230-9003-4591-8440-01f8a2ccf26e\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") "
Oct 30 22:41:04.037626 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.037559    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-trusted-ca-bundle\") pod \"6b5b9230-9003-4591-8440-01f8a2ccf26e\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") "
Oct 30 22:41:04.037626 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.037579    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b5b9230-9003-4591-8440-01f8a2ccf26e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6b5b9230-9003-4591-8440-01f8a2ccf26e-archives-path\") pod \"6b5b9230-9003-4591-8440-01f8a2ccf26e\" (UID: \"6b5b9230-9003-4591-8440-01f8a2ccf26e\") "
Oct 30 22:41:04.038203 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.038052    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-service-ca-bundle" pod "6b5b9230-9003-4591-8440-01f8a2ccf26e" (UID: "6b5b9230-9003-4591-8440-01f8a2ccf26e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:04.038203 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.038075    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-trusted-ca-bundle" pod "6b5b9230-9003-4591-8440-01f8a2ccf26e" (UID: "6b5b9230-9003-4591-8440-01f8a2ccf26e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:04.038203 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.038100    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5b9230-9003-4591-8440-01f8a2ccf26e-archives-path" pod "6b5b9230-9003-4591-8440-01f8a2ccf26e" (UID: "6b5b9230-9003-4591-8440-01f8a2ccf26e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:04.038355 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.038269    5060 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-trusted-ca-bundle\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:04.038355 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.038282    5060 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6b5b9230-9003-4591-8440-01f8a2ccf26e-archives-path\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:04.038355 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.038291    5060 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b5b9230-9003-4591-8440-01f8a2ccf26e-service-ca-bundle\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:04.040131 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.040113    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5b9230-9003-4591-8440-01f8a2ccf26e-kube-api-access-xf46t" pod "6b5b9230-9003-4591-8440-01f8a2ccf26e" (UID: "6b5b9230-9003-4591-8440-01f8a2ccf26e"). InnerVolumeSpecName "kube-api-access-xf46t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:04.139243 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.139198    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf46t\" (UniqueName: \"kubernetes.io/projected/6b5b9230-9003-4591-8440-01f8a2ccf26e-kube-api-access-xf46t\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:04.573134 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.573090    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mxxhw-rrcns" event={"ID":"6b5b9230-9003-4591-8440-01f8a2ccf26e","Type":"ContainerDied","Data":"985e2941dd43295049b8fa7f3568350a9e12bda39478591abe02a7a34a01240e"}
Oct 30 22:41:04.573797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.573774    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="985e2941dd43295049b8fa7f3568350a9e12bda39478591abe02a7a34a01240e"
Oct 30 22:41:04.573946 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:04.573533    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mxxhw-rrcns"
Oct 30 22:41:04.839325 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:41:04.839244    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:41:10.205259 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:10.205217    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:10.217045 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:10.217023    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:10.647114 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:10.647027    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:34.775871 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:41:34.775834    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:41:35.137079 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.136990    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"]
Oct 30 22:41:35.147414 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:41:35.147377    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024" cmd=["/bin/sleep","15"]
Oct 30 22:41:35.147612 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.147429    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:41:35.147612 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:41:35.147443    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t" podUID="96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" containerName="main" containerID="cri-o://e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024"
Oct 30 22:41:35.147612 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.147490    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t" podUID="96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" containerName="main" containerID="cri-o://e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024" gracePeriod=60
Oct 30 22:41:35.508161 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.508136    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:35.688990 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.688905    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tmp-dir\") pod \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") "
Oct 30 22:41:35.688990 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.688960    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-home\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-home\") pod \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") "
Oct 30 22:41:35.689195 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.689003    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-kube-api-access-rq9z5\" (UniqueName: \"kubernetes.io/projected/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-kube-api-access-rq9z5\") pod \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") "
Oct 30 22:41:35.689195 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.689038    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tls-certs\") pod \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") "
Oct 30 22:41:35.689195 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.689074    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-dshm\") pod \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") "
Oct 30 22:41:35.689195 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.689148    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-model-cache\") pod \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\" (UID: \"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4\") "
Oct 30 22:41:35.689329 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.689225    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tmp-dir" pod "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" (UID: "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.689329 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.689290    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-home" pod "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" (UID: "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.689526 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.689481    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-model-cache" pod "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" (UID: "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.689680 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.689528    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.689680 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.689549    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.691419 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.691393    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tls-certs" pod "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" (UID: "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:35.691537 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.691437    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-dshm" pod "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" (UID: "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.691537 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.691490    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-kube-api-access-rq9z5" pod "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" (UID: "96e56ebc-d0c0-4d96-9e3b-665f5de81bf4"). InnerVolumeSpecName "kube-api-access-rq9z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:35.754298 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.754265    5060 generic.go:356] "Generic (PLEG): container finished" podID="96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" containerID="e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024" exitCode=0
Oct 30 22:41:35.754469 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.754311    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"
Oct 30 22:41:35.754469 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.754347    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t" event={"ID":"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4","Type":"ContainerDied","Data":"e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024"}
Oct 30 22:41:35.754469 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.754403    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t" event={"ID":"96e56ebc-d0c0-4d96-9e3b-665f5de81bf4","Type":"ContainerDied","Data":"b113106c4424ee27c40b025a0e19813e7e9831f56ff9b5ebdc961df4627abb44"}
Oct 30 22:41:35.754469 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.754428    5060 scope.go:122] "RemoveContainer" containerID="e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024"
Oct 30 22:41:35.766546 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.766521    5060 scope.go:122] "RemoveContainer" containerID="e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024"
Oct 30 22:41:35.766856 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:41:35.766840    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024\": container with ID starting with e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024 not found: ID does not exist" containerID="e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024"
Oct 30 22:41:35.766910 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.766866    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024"} err="failed to get container status \"e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024\": rpc error: code = NotFound desc = could not find container \"e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024\": container with ID starting with e909e35d9889074e11c1e23fab9950701eb4d7ff26f40cb6b99776d7692b5024 not found: ID does not exist"
Oct 30 22:41:35.777705 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.777678    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"]
Oct 30 22:41:35.782565 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.782540    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-6667fc56b8-znp6t"]
Oct 30 22:41:35.789960 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.789924    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.789960 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.789953    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq9z5\" (UniqueName: \"kubernetes.io/projected/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-kube-api-access-rq9z5\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.790105 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.789968    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.790105 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:35.789980    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:37.289387 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:41:37.289347    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" path="/var/lib/kubelet/pods/96e56ebc-d0c0-4d96-9e3b-665f5de81bf4/volumes"
Oct 30 22:42:04.775098 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:42:04.775016    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:42:32.131376 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.131338    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"]
Oct 30 22:42:32.131840 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.131780    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6b5b9230-9003-4591-8440-01f8a2ccf26e" containerName="insights-gathering"
Oct 30 22:42:32.131840 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.131794    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5b9230-9003-4591-8440-01f8a2ccf26e" containerName="insights-gathering"
Oct 30 22:42:32.131840 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.131807    5060 cpu_manager.go:414] "containerMap: removing container" podUID="96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" containerName="main"
Oct 30 22:42:32.131840 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.131812    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" containerName="main"
Oct 30 22:42:32.131988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.131883    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5b9230-9003-4591-8440-01f8a2ccf26e" containerName="insights-gathering"
Oct 30 22:42:32.131988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.131891    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e56ebc-d0c0-4d96-9e3b-665f5de81bf4" containerName="main"
Oct 30 22:42:32.134832 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.134812    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.141856 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.141786    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Oct 30 22:42:32.141965 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.141940    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-vpmlk\""
Oct 30 22:42:32.150007 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.149981    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Oct 30 22:42:32.150199 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.150181    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Oct 30 22:42:32.151169 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.151146    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"]
Oct 30 22:42:32.286236 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.286202    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.286419 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.286247    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.286419 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.286269    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.286419 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.286369    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.286609 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.286436    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psbgk\" (UniqueName: \"kubernetes.io/projected/b47f032a-0612-461c-89f2-fdf0c5a05e2f-kube-api-access-psbgk\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.286609 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.286522    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.388028 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.387947    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.388028 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.387989    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psbgk\" (UniqueName: \"kubernetes.io/projected/b47f032a-0612-461c-89f2-fdf0c5a05e2f-kube-api-access-psbgk\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.388028 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.388027    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.388290 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.388082    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.388290 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.388132    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.388290 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.388164    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.388883 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.388524    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.388883 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.388527    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.388883 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.388736    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.390306 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.390277    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.390808 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.390778    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.407597 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.407563    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psbgk\" (UniqueName: \"kubernetes.io/projected/b47f032a-0612-461c-89f2-fdf0c5a05e2f-kube-api-access-psbgk\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.447653 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.447492    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:32.737147 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:32.737120    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"]
Oct 30 22:42:32.739221 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:42:32.739194    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47f032a_0612_461c_89f2_fdf0c5a05e2f.slice/crio-dc9030e61f8f0fbfa1908e6af5891962f614a8803dc420e0b4453a1f7a36cc36 WatchSource:0}: Error finding container dc9030e61f8f0fbfa1908e6af5891962f614a8803dc420e0b4453a1f7a36cc36: Status 404 returned error can't find the container with id dc9030e61f8f0fbfa1908e6af5891962f614a8803dc420e0b4453a1f7a36cc36
Oct 30 22:42:33.023590 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:33.023456    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv" event={"ID":"b47f032a-0612-461c-89f2-fdf0c5a05e2f","Type":"ContainerStarted","Data":"77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b"}
Oct 30 22:42:33.023590 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:33.023544    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv" event={"ID":"b47f032a-0612-461c-89f2-fdf0c5a05e2f","Type":"ContainerStarted","Data":"dc9030e61f8f0fbfa1908e6af5891962f614a8803dc420e0b4453a1f7a36cc36"}
Oct 30 22:42:33.450666 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:33.450618    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:34.832596 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:42:34.832554    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:42:41.023309 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:41.023255    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" podUID="a5c727ac326546386befce51c4d99c6d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 22:42:42.448135 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:42.448085    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:42.460533 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:42.460496    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:42.482743 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:42.482697    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv" podStartSLOduration=10.482684357 podStartE2EDuration="10.482684357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:33.061545331 +0000 UTC m=+1433.881283191" watchObservedRunningTime="2026-10-30 22:42:42.482684357 +0000 UTC m=+1443.302422176"
Oct 30 22:42:43.080872 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:43.080844    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:42:59.887325 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:59.887288    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"]
Oct 30 22:42:59.890779 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:59.890757    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:42:59.898384 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:59.897985    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Oct 30 22:42:59.898972 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:59.898952    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Oct 30 22:42:59.898972 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:59.898960    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:42:59.899118 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:59.899097    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-d846j\""
Oct 30 22:42:59.909396 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:42:59.909362    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"]
Oct 30 22:43:00.012021 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.011981    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-certs\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.012198 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.012089    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-envoy\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.012198 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.012146    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-podinfo\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.012290 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.012240    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-socket\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.012290 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.012272    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-data\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.012424 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.012401    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-token\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.012492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.012441    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e86c614-f713-4c75-abdb-bdb9524004fa-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.012492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.012469    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzrk9\" (UniqueName: \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-kube-api-access-nzrk9\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.012604 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.012554    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-credential-socket\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113060 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113012    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-socket\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113060 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113066    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-data\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113315 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113120    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-token\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113315 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113146    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e86c614-f713-4c75-abdb-bdb9524004fa-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113315 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113174    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzrk9\" (UniqueName: \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-kube-api-access-nzrk9\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113315 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113206    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-credential-socket\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113315 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113245    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-certs\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113315 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113301    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-envoy\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113626 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113324    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-podinfo\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113626 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113474    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-socket\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113626 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113518    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-data\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113720 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113637    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-credential-socket\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113720 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113701    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-certs\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.113917 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.113893    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e86c614-f713-4c75-abdb-bdb9524004fa-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.115617 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.115599    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-envoy\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.115753 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.115738    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-podinfo\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.125867 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.125843    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-token\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.126007 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.125988    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzrk9\" (UniqueName: \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-kube-api-access-nzrk9\") pod \"router-gateway-1-openshift-default-9fcdc5ddf-h7hcs\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.205826 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.205795    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:00.490665 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.490640    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"]
Oct 30 22:43:00.493027 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:43:00.492986    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e86c614_f713_4c75_abdb_bdb9524004fa.slice/crio-bb78afedbefc473c4ad8c88b53d93d252bca274e55e6f852c54f5768d82f7a70 WatchSource:0}: Error finding container bb78afedbefc473c4ad8c88b53d93d252bca274e55e6f852c54f5768d82f7a70: Status 404 returned error can't find the container with id bb78afedbefc473c4ad8c88b53d93d252bca274e55e6f852c54f5768d82f7a70
Oct 30 22:43:00.495131 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.495094    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:43:00.495211 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.495188    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:43:00.495251 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:00.495230    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:43:01.151873 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:01.151837    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs" event={"ID":"1e86c614-f713-4c75-abdb-bdb9524004fa","Type":"ContainerStarted","Data":"64081721af0a1c88f07533d1cffa8454d1671ff734f87220b6104dc07d76e2d5"}
Oct 30 22:43:01.152308 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:01.151888    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs" event={"ID":"1e86c614-f713-4c75-abdb-bdb9524004fa","Type":"ContainerStarted","Data":"bb78afedbefc473c4ad8c88b53d93d252bca274e55e6f852c54f5768d82f7a70"}
Oct 30 22:43:01.178246 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:01.178197    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs" podStartSLOduration=2.178180196 podStartE2EDuration="2.178180196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:01.177355095 +0000 UTC m=+1461.997092913" watchObservedRunningTime="2026-10-30 22:43:01.178180196 +0000 UTC m=+1461.997918014"
Oct 30 22:43:01.205920 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:01.205887    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:01.210830 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:01.210805    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:02.156191 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:02.156156    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:02.157175 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:02.157159    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:04.778035 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:43:04.777999    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:43:13.076313 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.076276    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"]
Oct 30 22:43:13.086487 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:43:13.086438    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b" cmd=["/bin/sleep","15"]
Oct 30 22:43:13.086648 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.086519    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:43:13.086648 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:43:13.086540    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv" podUID="b47f032a-0612-461c-89f2-fdf0c5a05e2f" containerName="main" containerID="cri-o://77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b"
Oct 30 22:43:13.086648 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.086593    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv" podUID="b47f032a-0612-461c-89f2-fdf0c5a05e2f" containerName="main" containerID="cri-o://77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b" gracePeriod=60
Oct 30 22:43:13.429592 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.429562    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:43:13.534035 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.534002    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47f032a-0612-461c-89f2-fdf0c5a05e2f-kube-api-access-psbgk\" (UniqueName: \"kubernetes.io/projected/b47f032a-0612-461c-89f2-fdf0c5a05e2f-kube-api-access-psbgk\") pod \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") "
Oct 30 22:43:13.534228 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.534056    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-model-cache\") pod \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") "
Oct 30 22:43:13.534228 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.534083    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-home\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-home\") pod \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") "
Oct 30 22:43:13.534228 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.534128    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tls-certs\") pod \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") "
Oct 30 22:43:13.534228 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.534175    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-dshm\") pod \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") "
Oct 30 22:43:13.534446 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.534285    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tmp-dir\") pod \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\" (UID: \"b47f032a-0612-461c-89f2-fdf0c5a05e2f\") "
Oct 30 22:43:13.534446 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.534355    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-model-cache" pod "b47f032a-0612-461c-89f2-fdf0c5a05e2f" (UID: "b47f032a-0612-461c-89f2-fdf0c5a05e2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:13.534627 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.534589    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tmp-dir" pod "b47f032a-0612-461c-89f2-fdf0c5a05e2f" (UID: "b47f032a-0612-461c-89f2-fdf0c5a05e2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:13.534720 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.534615    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:13.534720 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.534630    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-home" pod "b47f032a-0612-461c-89f2-fdf0c5a05e2f" (UID: "b47f032a-0612-461c-89f2-fdf0c5a05e2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:13.536550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.536520    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tls-certs" pod "b47f032a-0612-461c-89f2-fdf0c5a05e2f" (UID: "b47f032a-0612-461c-89f2-fdf0c5a05e2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:13.536727 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.536708    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47f032a-0612-461c-89f2-fdf0c5a05e2f-kube-api-access-psbgk" pod "b47f032a-0612-461c-89f2-fdf0c5a05e2f" (UID: "b47f032a-0612-461c-89f2-fdf0c5a05e2f"). InnerVolumeSpecName "kube-api-access-psbgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:13.536786 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.536714    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-dshm" pod "b47f032a-0612-461c-89f2-fdf0c5a05e2f" (UID: "b47f032a-0612-461c-89f2-fdf0c5a05e2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:13.635311 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.635227    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:13.635311 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.635271    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psbgk\" (UniqueName: \"kubernetes.io/projected/b47f032a-0612-461c-89f2-fdf0c5a05e2f-kube-api-access-psbgk\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:13.635311 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.635288    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:13.635311 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.635302    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b47f032a-0612-461c-89f2-fdf0c5a05e2f-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:13.635583 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:13.635314    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b47f032a-0612-461c-89f2-fdf0c5a05e2f-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:14.214431 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:14.214393    5060 generic.go:356] "Generic (PLEG): container finished" podID="b47f032a-0612-461c-89f2-fdf0c5a05e2f" containerID="77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b" exitCode=0
Oct 30 22:43:14.214897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:14.214443    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"
Oct 30 22:43:14.214897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:14.214479    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv" event={"ID":"b47f032a-0612-461c-89f2-fdf0c5a05e2f","Type":"ContainerDied","Data":"77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b"}
Oct 30 22:43:14.214897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:14.214552    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv" event={"ID":"b47f032a-0612-461c-89f2-fdf0c5a05e2f","Type":"ContainerDied","Data":"dc9030e61f8f0fbfa1908e6af5891962f614a8803dc420e0b4453a1f7a36cc36"}
Oct 30 22:43:14.214897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:14.214576    5060 scope.go:122] "RemoveContainer" containerID="77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b"
Oct 30 22:43:14.224216 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:14.224194    5060 scope.go:122] "RemoveContainer" containerID="77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b"
Oct 30 22:43:14.224541 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:43:14.224500    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b\": container with ID starting with 77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b not found: ID does not exist" containerID="77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b"
Oct 30 22:43:14.224615 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:14.224548    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b"} err="failed to get container status \"77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b\": rpc error: code = NotFound desc = could not find container \"77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b\": container with ID starting with 77c66135789edc3ceb3121b01ecfc5e662cb08a26e37b1f7e10546ecb8bcdd3b not found: ID does not exist"
Oct 30 22:43:14.238877 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:14.238853    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"]
Oct 30 22:43:14.243789 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:14.243770    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-k89sv"]
Oct 30 22:43:15.221089 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:15.221043    5060 generic.go:356] "Generic (PLEG): container finished" podID="1e86c614-f713-4c75-abdb-bdb9524004fa" containerID="64081721af0a1c88f07533d1cffa8454d1671ff734f87220b6104dc07d76e2d5" exitCode=0
Oct 30 22:43:15.221590 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:15.221106    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs" event={"ID":"1e86c614-f713-4c75-abdb-bdb9524004fa","Type":"ContainerDied","Data":"64081721af0a1c88f07533d1cffa8454d1671ff734f87220b6104dc07d76e2d5"}
Oct 30 22:43:15.221590 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:15.221533    5060 scope.go:122] "RemoveContainer" containerID="64081721af0a1c88f07533d1cffa8454d1671ff734f87220b6104dc07d76e2d5"
Oct 30 22:43:15.222564 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:15.222530    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:43:15.222635 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:15.222618    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:43:15.222673 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:15.222663    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:43:15.295392 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:15.292705    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b47f032a-0612-461c-89f2-fdf0c5a05e2f" path="/var/lib/kubelet/pods/b47f032a-0612-461c-89f2-fdf0c5a05e2f/volumes"
Oct 30 22:43:16.227677 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:16.227646    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs" event={"ID":"1e86c614-f713-4c75-abdb-bdb9524004fa","Type":"ContainerStarted","Data":"20f281b667c7918cff989407d62c6bba487add3804c759e14094e423f38a7bd5"}
Oct 30 22:43:17.156079 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:17.156042    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:17.206669 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:17.206639    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:17.211312 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:17.211287    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:17.232876 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:17.232848    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:43:34.775035 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:43:34.774997    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:43:39.421880 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:39.421846    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:43:39.424912 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:43:39.424881    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:44:04.775312 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:44:04.775274    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:44:05.892168 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.892131    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"]
Oct 30 22:44:05.892701 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.892565    5060 cpu_manager.go:414] "containerMap: removing container" podUID="b47f032a-0612-461c-89f2-fdf0c5a05e2f" containerName="main"
Oct 30 22:44:05.892701 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.892579    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47f032a-0612-461c-89f2-fdf0c5a05e2f" containerName="main"
Oct 30 22:44:05.892701 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.892658    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="b47f032a-0612-461c-89f2-fdf0c5a05e2f" containerName="main"
Oct 30 22:44:05.895727 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.895707    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:05.906437 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.906411    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:05.906569 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.906465    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:05.906569 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.906557    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:05.906683 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.906591    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:05.906683 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.906610    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85mp8\" (UniqueName: \"kubernetes.io/projected/bd32575f-e74f-45b9-9c57-6ba3a487c72b-kube-api-access-85mp8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:05.906756 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.906702    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:05.912979 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.912957    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"]
Oct 30 22:44:05.913326 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.913311    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Oct 30 22:44:05.914805 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.914779    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Oct 30 22:44:05.932734 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.932710    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-b4s4l\""
Oct 30 22:44:05.933378 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:05.933359    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Oct 30 22:44:06.007177 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.007139    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.007382 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.007192    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.007382 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.007230    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.007382 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.007254    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.007382 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.007273    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85mp8\" (UniqueName: \"kubernetes.io/projected/bd32575f-e74f-45b9-9c57-6ba3a487c72b-kube-api-access-85mp8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.007382 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.007295    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.007731 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.007697    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.007853 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.007748    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.007853 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.007782    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.009442 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.009422    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.009710 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.009694    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.020210 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.020181    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85mp8\" (UniqueName: \"kubernetes.io/projected/bd32575f-e74f-45b9-9c57-6ba3a487c72b-kube-api-access-85mp8\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.208109 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.208073    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:06.484847 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:06.484822    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"]
Oct 30 22:44:06.487029 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:44:06.487003    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd32575f_e74f_45b9_9c57_6ba3a487c72b.slice/crio-f8576826b49260b7162dbb9d908b780ccce2c2abc50bb51bce1e71fd42c3bc47 WatchSource:0}: Error finding container f8576826b49260b7162dbb9d908b780ccce2c2abc50bb51bce1e71fd42c3bc47: Status 404 returned error can't find the container with id f8576826b49260b7162dbb9d908b780ccce2c2abc50bb51bce1e71fd42c3bc47
Oct 30 22:44:07.508739 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:07.508700    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b" event={"ID":"bd32575f-e74f-45b9-9c57-6ba3a487c72b","Type":"ContainerStarted","Data":"fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a"}
Oct 30 22:44:07.510528 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:07.509310    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b" event={"ID":"bd32575f-e74f-45b9-9c57-6ba3a487c72b","Type":"ContainerStarted","Data":"f8576826b49260b7162dbb9d908b780ccce2c2abc50bb51bce1e71fd42c3bc47"}
Oct 30 22:44:07.536501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:07.536450    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b" podStartSLOduration=2.536435768 podStartE2EDuration="2.536435768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:07.535793509 +0000 UTC m=+1528.355531330" watchObservedRunningTime="2026-10-30 22:44:07.536435768 +0000 UTC m=+1528.356173586"
Oct 30 22:44:08.208738 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:08.208702    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:16.208606 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:16.208570    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:16.220839 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:16.220814    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:16.583313 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:16.583236    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:34.775867 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:44:34.775826    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:44:41.271386 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.271353    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"]
Oct 30 22:44:41.281167 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:44:41.281134    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a" cmd=["/bin/sleep","15"]
Oct 30 22:44:41.281312 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.281182    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:44:41.281312 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:44:41.281195    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b" podUID="bd32575f-e74f-45b9-9c57-6ba3a487c72b" containerName="main" containerID="cri-o://fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a"
Oct 30 22:44:41.281312 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.281229    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b" podUID="bd32575f-e74f-45b9-9c57-6ba3a487c72b" containerName="main" containerID="cri-o://fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a" gracePeriod=60
Oct 30 22:44:41.543071 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.543043    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:41.685614 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.685570    5060 generic.go:356] "Generic (PLEG): container finished" podID="bd32575f-e74f-45b9-9c57-6ba3a487c72b" containerID="fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a" exitCode=0
Oct 30 22:44:41.685614 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.685614    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"
Oct 30 22:44:41.685815 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.685650    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b" event={"ID":"bd32575f-e74f-45b9-9c57-6ba3a487c72b","Type":"ContainerDied","Data":"fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a"}
Oct 30 22:44:41.685815 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.685699    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b" event={"ID":"bd32575f-e74f-45b9-9c57-6ba3a487c72b","Type":"ContainerDied","Data":"f8576826b49260b7162dbb9d908b780ccce2c2abc50bb51bce1e71fd42c3bc47"}
Oct 30 22:44:41.685815 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.685722    5060 scope.go:122] "RemoveContainer" containerID="fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a"
Oct 30 22:44:41.694381 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.694360    5060 scope.go:122] "RemoveContainer" containerID="fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a"
Oct 30 22:44:41.694742 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:44:41.694721    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a\": container with ID starting with fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a not found: ID does not exist" containerID="fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a"
Oct 30 22:44:41.694849 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.694752    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a"} err="failed to get container status \"fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a\": rpc error: code = NotFound desc = could not find container \"fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a\": container with ID starting with fc688d35a81371b280c1c9551cd57e6e96ed81a3f944f003997339f238319b3a not found: ID does not exist"
Oct 30 22:44:41.702029 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.702008    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd32575f-e74f-45b9-9c57-6ba3a487c72b-kube-api-access-85mp8\" (UniqueName: \"kubernetes.io/projected/bd32575f-e74f-45b9-9c57-6ba3a487c72b-kube-api-access-85mp8\") pod \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") "
Oct 30 22:44:41.702133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.702040    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-home\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-home\") pod \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") "
Oct 30 22:44:41.702133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.702087    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-dshm\") pod \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") "
Oct 30 22:44:41.702133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.702111    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tls-certs\") pod \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") "
Oct 30 22:44:41.702133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.702128    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tmp-dir\") pod \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") "
Oct 30 22:44:41.702395 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.702199    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-model-cache\") pod \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\" (UID: \"bd32575f-e74f-45b9-9c57-6ba3a487c72b\") "
Oct 30 22:44:41.702612 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.702559    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-model-cache" pod "bd32575f-e74f-45b9-9c57-6ba3a487c72b" (UID: "bd32575f-e74f-45b9-9c57-6ba3a487c72b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:41.702612 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.702499    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-home" pod "bd32575f-e74f-45b9-9c57-6ba3a487c72b" (UID: "bd32575f-e74f-45b9-9c57-6ba3a487c72b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:41.702612 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.702565    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tmp-dir" pod "bd32575f-e74f-45b9-9c57-6ba3a487c72b" (UID: "bd32575f-e74f-45b9-9c57-6ba3a487c72b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:41.704357 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.704325    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tls-certs" pod "bd32575f-e74f-45b9-9c57-6ba3a487c72b" (UID: "bd32575f-e74f-45b9-9c57-6ba3a487c72b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:41.704456 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.704361    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd32575f-e74f-45b9-9c57-6ba3a487c72b-kube-api-access-85mp8" pod "bd32575f-e74f-45b9-9c57-6ba3a487c72b" (UID: "bd32575f-e74f-45b9-9c57-6ba3a487c72b"). InnerVolumeSpecName "kube-api-access-85mp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:41.704456 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.704403    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-dshm" pod "bd32575f-e74f-45b9-9c57-6ba3a487c72b" (UID: "bd32575f-e74f-45b9-9c57-6ba3a487c72b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:41.803550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.803444    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:41.803550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.803474    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85mp8\" (UniqueName: \"kubernetes.io/projected/bd32575f-e74f-45b9-9c57-6ba3a487c72b-kube-api-access-85mp8\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:41.803550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.803483    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:41.803550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.803492    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:41.803550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.803500    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:41.803550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:41.803533    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd32575f-e74f-45b9-9c57-6ba3a487c72b-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:42.009114 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:42.009083    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"]
Oct 30 22:44:42.013711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:42.013686    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-7776l9b6b"]
Oct 30 22:44:43.289165 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:44:43.289127    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd32575f-e74f-45b9-9c57-6ba3a487c72b" path="/var/lib/kubelet/pods/bd32575f-e74f-45b9-9c57-6ba3a487c72b/volumes"
Oct 30 22:45:04.774706 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:45:04.774630    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:45:23.733192 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:23.733154    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"]
Oct 30 22:45:23.733787 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:23.733348    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs" podUID="1e86c614-f713-4c75-abdb-bdb9524004fa" containerName="istio-proxy" containerID="cri-o://20f281b667c7918cff989407d62c6bba487add3804c759e14094e423f38a7bd5" gracePeriod=30
Oct 30 22:45:24.225572 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.225535    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"]
Oct 30 22:45:24.226141 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.226115    5060 cpu_manager.go:414] "containerMap: removing container" podUID="bd32575f-e74f-45b9-9c57-6ba3a487c72b" containerName="main"
Oct 30 22:45:24.226141 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.226139    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd32575f-e74f-45b9-9c57-6ba3a487c72b" containerName="main"
Oct 30 22:45:24.226294 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.226215    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd32575f-e74f-45b9-9c57-6ba3a487c72b" containerName="main"
Oct 30 22:45:24.229963 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.229943    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.237059 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.236633    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:45:24.237059 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.236794    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Oct 30 22:45:24.237059 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.236865    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Oct 30 22:45:24.238162 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.238139    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Oct 30 22:45:24.238292 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.238274    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Oct 30 22:45:24.238376 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.238322    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-5q6lw\""
Oct 30 22:45:24.242991 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.242970    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"]
Oct 30 22:45:24.368254 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.368218    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-cabundle-cert\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.368254 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.368259    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kserve-provision-location\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.368459 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.368286    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl5jx\" (UniqueName: \"kubernetes.io/projected/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kube-api-access-zl5jx\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.368459 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.368387    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-home\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.368549 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.368477    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tls-certs\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.368549 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.368534    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tmp-dir\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.368612 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.368559    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-model-cache\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.368612 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.368579    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-dshm\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.469897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.469863    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tls-certs\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.470066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.469911    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tmp-dir\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.470066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.469940    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-model-cache\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.470066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.469964    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-dshm\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.470066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.469989    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-cabundle-cert\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.470066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.470017    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kserve-provision-location\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.470066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.470054    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl5jx\" (UniqueName: \"kubernetes.io/projected/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kube-api-access-zl5jx\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.470268 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.470082    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-home\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.470453 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.470433    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-home\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.470874 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.470846    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kserve-provision-location\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.471046 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.470941    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-model-cache\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.471173 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.471054    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-cabundle-cert\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.471306 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.471231    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tmp-dir\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.472778 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.472757    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tls-certs\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.473333 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.473312    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-dshm\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.487965 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.487898    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl5jx\" (UniqueName: \"kubernetes.io/projected/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kube-api-access-zl5jx\") pod \"auth-enabled-test-kserve-65ddcfbf9c-wjgsw\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.546675 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.546638    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:24.831847 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.831818    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"]
Oct 30 22:45:24.833927 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:45:24.833893    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa41b689_d2dc_401d_8f82_f87fb8e1ad10.slice/crio-6c578880dd5853fe0045ecc6a8ed06dc670295fc53c3b16e892c6a4da61a8409 WatchSource:0}: Error finding container 6c578880dd5853fe0045ecc6a8ed06dc670295fc53c3b16e892c6a4da61a8409: Status 404 returned error can't find the container with id 6c578880dd5853fe0045ecc6a8ed06dc670295fc53c3b16e892c6a4da61a8409
Oct 30 22:45:24.884010 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:24.883976    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" event={"ID":"aa41b689-d2dc-401d-8f82-f87fb8e1ad10","Type":"ContainerStarted","Data":"6c578880dd5853fe0045ecc6a8ed06dc670295fc53c3b16e892c6a4da61a8409"}
Oct 30 22:45:25.890248 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:25.890212    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" event={"ID":"aa41b689-d2dc-401d-8f82-f87fb8e1ad10","Type":"ContainerStarted","Data":"eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff"}
Oct 30 22:45:27.901392 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:27.901360    5060 generic.go:356] "Generic (PLEG): container finished" podID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerID="eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff" exitCode=0
Oct 30 22:45:27.901802 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:27.901437    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" event={"ID":"aa41b689-d2dc-401d-8f82-f87fb8e1ad10","Type":"ContainerDied","Data":"eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff"}
Oct 30 22:45:28.910793 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:28.910755    5060 generic.go:356] "Generic (PLEG): container finished" podID="1e86c614-f713-4c75-abdb-bdb9524004fa" containerID="20f281b667c7918cff989407d62c6bba487add3804c759e14094e423f38a7bd5" exitCode=0
Oct 30 22:45:28.911228 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:28.910830    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs" event={"ID":"1e86c614-f713-4c75-abdb-bdb9524004fa","Type":"ContainerDied","Data":"20f281b667c7918cff989407d62c6bba487add3804c759e14094e423f38a7bd5"}
Oct 30 22:45:28.911228 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:28.910911    5060 scope.go:122] "RemoveContainer" containerID="64081721af0a1c88f07533d1cffa8454d1671ff734f87220b6104dc07d76e2d5"
Oct 30 22:45:28.915772 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:28.915110    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" event={"ID":"aa41b689-d2dc-401d-8f82-f87fb8e1ad10","Type":"ContainerStarted","Data":"74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e"}
Oct 30 22:45:28.945183 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:28.945132    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podStartSLOduration=2.5939042839999997 podStartE2EDuration="4.945119951s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.351215667s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:28.943897291 +0000 UTC m=+1609.763635111" watchObservedRunningTime="2026-10-30 22:45:28.945119951 +0000 UTC m=+1609.764857769"
Oct 30 22:45:28.982797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:28.982766    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:45:29.118826 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.118796    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-socket\") pod \"1e86c614-f713-4c75-abdb-bdb9524004fa\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") "
Oct 30 22:45:29.119083 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.118831    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e86c614-f713-4c75-abdb-bdb9524004fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e86c614-f713-4c75-abdb-bdb9524004fa-istiod-ca-cert\") pod \"1e86c614-f713-4c75-abdb-bdb9524004fa\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") "
Oct 30 22:45:29.119083 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.118861    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-token\" (UniqueName: \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-token\") pod \"1e86c614-f713-4c75-abdb-bdb9524004fa\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") "
Oct 30 22:45:29.119083 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.118904    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-envoy\") pod \"1e86c614-f713-4c75-abdb-bdb9524004fa\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") "
Oct 30 22:45:29.119083 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.118943    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-podinfo\") pod \"1e86c614-f713-4c75-abdb-bdb9524004fa\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") "
Oct 30 22:45:29.119083 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.119020    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-data\") pod \"1e86c614-f713-4c75-abdb-bdb9524004fa\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") "
Oct 30 22:45:29.119083 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.119057    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-credential-socket\") pod \"1e86c614-f713-4c75-abdb-bdb9524004fa\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") "
Oct 30 22:45:29.119396 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.119113    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-socket" pod "1e86c614-f713-4c75-abdb-bdb9524004fa" (UID: "1e86c614-f713-4c75-abdb-bdb9524004fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:29.119396 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.119122    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-certs\") pod \"1e86c614-f713-4c75-abdb-bdb9524004fa\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") "
Oct 30 22:45:29.119396 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.119211    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-kube-api-access-nzrk9\" (UniqueName: \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-kube-api-access-nzrk9\") pod \"1e86c614-f713-4c75-abdb-bdb9524004fa\" (UID: \"1e86c614-f713-4c75-abdb-bdb9524004fa\") "
Oct 30 22:45:29.119616 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.119599    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.119687 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.119635    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-data" pod "1e86c614-f713-4c75-abdb-bdb9524004fa" (UID: "1e86c614-f713-4c75-abdb-bdb9524004fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:29.119687 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.119666    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-certs" pod "1e86c614-f713-4c75-abdb-bdb9524004fa" (UID: "1e86c614-f713-4c75-abdb-bdb9524004fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:29.119796 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.119681    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-credential-socket" pod "1e86c614-f713-4c75-abdb-bdb9524004fa" (UID: "1e86c614-f713-4c75-abdb-bdb9524004fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:29.119901 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.119878    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e86c614-f713-4c75-abdb-bdb9524004fa-istiod-ca-cert" pod "1e86c614-f713-4c75-abdb-bdb9524004fa" (UID: "1e86c614-f713-4c75-abdb-bdb9524004fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:29.121400 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.121370    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-envoy" pod "1e86c614-f713-4c75-abdb-bdb9524004fa" (UID: "1e86c614-f713-4c75-abdb-bdb9524004fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:29.121538 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.121495    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-token" pod "1e86c614-f713-4c75-abdb-bdb9524004fa" (UID: "1e86c614-f713-4c75-abdb-bdb9524004fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:29.121774 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.121746    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-kube-api-access-nzrk9" pod "1e86c614-f713-4c75-abdb-bdb9524004fa" (UID: "1e86c614-f713-4c75-abdb-bdb9524004fa"). InnerVolumeSpecName "kube-api-access-nzrk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:29.121858 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.121775    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-podinfo" pod "1e86c614-f713-4c75-abdb-bdb9524004fa" (UID: "1e86c614-f713-4c75-abdb-bdb9524004fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:45:29.220047 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.220014    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.220047 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.220041    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzrk9\" (UniqueName: \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-kube-api-access-nzrk9\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.220047 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.220051    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e86c614-f713-4c75-abdb-bdb9524004fa-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.220282 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.220059    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.220282 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.220068    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.220282 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.220076    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.220282 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.220083    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.220282 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.220091    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e86c614-f713-4c75-abdb-bdb9524004fa-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:29.547775 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.547681    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:29.922566 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.922437    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"
Oct 30 22:45:29.922566 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.922468    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs" event={"ID":"1e86c614-f713-4c75-abdb-bdb9524004fa","Type":"ContainerDied","Data":"bb78afedbefc473c4ad8c88b53d93d252bca274e55e6f852c54f5768d82f7a70"}
Oct 30 22:45:29.922566 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.922545    5060 scope.go:122] "RemoveContainer" containerID="20f281b667c7918cff989407d62c6bba487add3804c759e14094e423f38a7bd5"
Oct 30 22:45:29.944712 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.944675    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"]
Oct 30 22:45:29.949153 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:29.949128    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-9fcdc5ddf-h7hcs"]
Oct 30 22:45:31.290008 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:31.289966    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e86c614-f713-4c75-abdb-bdb9524004fa" path="/var/lib/kubelet/pods/1e86c614-f713-4c75-abdb-bdb9524004fa/volumes"
Oct 30 22:45:33.744646 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.744601    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"]
Oct 30 22:45:33.745153 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.745006    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1e86c614-f713-4c75-abdb-bdb9524004fa" containerName="istio-proxy"
Oct 30 22:45:33.745153 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.745019    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e86c614-f713-4c75-abdb-bdb9524004fa" containerName="istio-proxy"
Oct 30 22:45:33.745153 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.745046    5060 cpu_manager.go:414] "containerMap: removing container" podUID="1e86c614-f713-4c75-abdb-bdb9524004fa" containerName="istio-proxy"
Oct 30 22:45:33.745153 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.745051    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e86c614-f713-4c75-abdb-bdb9524004fa" containerName="istio-proxy"
Oct 30 22:45:33.745153 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.745114    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e86c614-f713-4c75-abdb-bdb9524004fa" containerName="istio-proxy"
Oct 30 22:45:33.745153 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.745125    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e86c614-f713-4c75-abdb-bdb9524004fa" containerName="istio-proxy"
Oct 30 22:45:33.749253 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.749227    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.757000 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.756971    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Oct 30 22:45:33.759943 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.759887    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:45:33.762704 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.762678    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Oct 30 22:45:33.762704 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.762678    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Oct 30 22:45:33.765924 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.765895    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"]
Oct 30 22:45:33.856675 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.856643    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-jwx64\""
Oct 30 22:45:33.856675 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.856674    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Oct 30 22:45:33.860329 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.860298    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-dshm\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.860486 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.860363    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.860486 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.860407    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-model-cache\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.860486 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.860432    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tls-certs\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.860486 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.860468    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.860710 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.860501    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-home\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.860710 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.860584    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.860710 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.860620    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866kx\" (UniqueName: \"kubernetes.io/projected/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kube-api-access-866kx\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.961338 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961293    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.961537 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961353    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866kx\" (UniqueName: \"kubernetes.io/projected/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kube-api-access-866kx\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.961537 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961405    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-dshm\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.961537 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961450    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.961537 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961490    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-model-cache\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.961750 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961727    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.961804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961681    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tls-certs\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.961857 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961842    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.961914 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961877    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-model-cache\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.961914 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961879    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-home\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.962015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.961922    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.962137 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.962113    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-home\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.962455 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.962434    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.963723 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.963704    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-dshm\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.964096 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.964076    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tls-certs\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:33.977200 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:33.977165    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-866kx\" (UniqueName: \"kubernetes.io/projected/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kube-api-access-866kx\") pod \"tokenizer-clean-path-test-kserve-79d7476f4b-v284m\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:34.067482 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.067378    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:34.106004 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.105968    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"]
Oct 30 22:45:34.111501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.111471    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.119118 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.119076    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"]
Oct 30 22:45:34.265250 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.265197    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74a8a072-3630-4833-a205-7d61e5cfe292-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.265250 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.265249    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a8a072-3630-4833-a205-7d61e5cfe292-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.265501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.265325    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.265501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.265403    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.366374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.366116    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.366374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.366237    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.366374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.366296    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74a8a072-3630-4833-a205-7d61e5cfe292-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.366374 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.366320    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a8a072-3630-4833-a205-7d61e5cfe292-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.366652 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.366602    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.366652 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.366641    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.367032 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.367013    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74a8a072-3630-4833-a205-7d61e5cfe292-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.367137 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.367116    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"]
Oct 30 22:45:34.369219 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.369195    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a8a072-3630-4833-a205-7d61e5cfe292-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.369736 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:45:34.369715    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534c07e4_2a3c_4bcd_bc66_77f81d47b1b7.slice/crio-92d14e96ed32fa07232a8c8f19d73bba1484ccf78759ad62a3292bde8916f125 WatchSource:0}: Error finding container 92d14e96ed32fa07232a8c8f19d73bba1484ccf78759ad62a3292bde8916f125: Status 404 returned error can't find the container with id 92d14e96ed32fa07232a8c8f19d73bba1484ccf78759ad62a3292bde8916f125
Oct 30 22:45:34.440056 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.440027    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:34.547785 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.547600    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:45:34.549058 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.549027    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:45:34.738650 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.738618    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"]
Oct 30 22:45:34.741170 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:45:34.741142    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a8a072_3630_4833_a205_7d61e5cfe292.slice/crio-21aa8b4b719b865dd912f483c6b354522a21b9a4e2a7bc69ed7708729d806ded WatchSource:0}: Error finding container 21aa8b4b719b865dd912f483c6b354522a21b9a4e2a7bc69ed7708729d806ded: Status 404 returned error can't find the container with id 21aa8b4b719b865dd912f483c6b354522a21b9a4e2a7bc69ed7708729d806ded
Oct 30 22:45:34.780502 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:45:34.780464    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:45:34.954468 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.954432    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" event={"ID":"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7","Type":"ContainerStarted","Data":"621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5"}
Oct 30 22:45:34.954663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.954491    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" event={"ID":"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7","Type":"ContainerStarted","Data":"92d14e96ed32fa07232a8c8f19d73bba1484ccf78759ad62a3292bde8916f125"}
Oct 30 22:45:34.956527 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.956468    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q" event={"ID":"74a8a072-3630-4833-a205-7d61e5cfe292","Type":"ContainerStarted","Data":"8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a"}
Oct 30 22:45:34.956688 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:34.956545    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q" event={"ID":"74a8a072-3630-4833-a205-7d61e5cfe292","Type":"ContainerStarted","Data":"21aa8b4b719b865dd912f483c6b354522a21b9a4e2a7bc69ed7708729d806ded"}
Oct 30 22:45:37.985341 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:37.985304    5060 generic.go:356] "Generic (PLEG): container finished" podID="74a8a072-3630-4833-a205-7d61e5cfe292" containerID="8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a" exitCode=0
Oct 30 22:45:37.985844 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:37.985378    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q" event={"ID":"74a8a072-3630-4833-a205-7d61e5cfe292","Type":"ContainerDied","Data":"8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a"}
Oct 30 22:45:37.987039 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:37.987014    5060 generic.go:356] "Generic (PLEG): container finished" podID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" containerID="621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5" exitCode=0
Oct 30 22:45:37.987193 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:37.987046    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" event={"ID":"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7","Type":"ContainerDied","Data":"621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5"}
Oct 30 22:45:38.994574 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:38.994535    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" event={"ID":"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7","Type":"ContainerStarted","Data":"62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405"}
Oct 30 22:45:38.996521 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:38.996481    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q" event={"ID":"74a8a072-3630-4833-a205-7d61e5cfe292","Type":"ContainerStarted","Data":"e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654"}
Oct 30 22:45:38.996708 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:38.996687    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:45:39.020974 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:39.020919    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" podStartSLOduration=3.299984441 podStartE2EDuration="6.020897785s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.720913344s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:39.020743481 +0000 UTC m=+1619.840481300" watchObservedRunningTime="2026-10-30 22:45:39.020897785 +0000 UTC m=+1619.840635603"
Oct 30 22:45:39.043559 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:39.043485    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q" podStartSLOduration=2.346033251 podStartE2EDuration="5.043471346s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.697438095s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:39.042797456 +0000 UTC m=+1619.862535275" watchObservedRunningTime="2026-10-30 22:45:39.043471346 +0000 UTC m=+1619.863209164"
Oct 30 22:45:39.068421 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:39.068385    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:44.067890 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:44.067849    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:44.080422 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:44.080388    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:44.547885 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:44.547844    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:45:45.042502 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:45.042463    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:45:54.547353 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:45:54.547292    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:46:04.547501 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:04.547454    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:46:04.775485 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:46:04.775443    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:46:10.012853 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:10.012821    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:46:11.520056 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:11.520029    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q_74a8a072-3630-4833-a205-7d61e5cfe292/main/0.log"
Oct 30 22:46:11.752285 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:11.752250    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"]
Oct 30 22:46:11.755327 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:11.755299    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"]
Oct 30 22:46:11.755533 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:11.755491    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q" podUID="74a8a072-3630-4833-a205-7d61e5cfe292" containerName="main" containerID="cri-o://e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654" gracePeriod=30
Oct 30 22:46:11.763095 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:46:11.763055    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405" cmd=["/bin/sleep","15"]
Oct 30 22:46:11.763196 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:11.763111    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:46:11.763196 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:46:11.763129    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" podUID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" containerName="main" containerID="cri-o://62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405"
Oct 30 22:46:11.763196 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:11.763169    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" podUID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" containerName="main" containerID="cri-o://62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405" gracePeriod=60
Oct 30 22:46:12.119800 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.119733    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:46:12.171994 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.171961    5060 generic.go:356] "Generic (PLEG): container finished" podID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" containerID="62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405" exitCode=0
Oct 30 22:46:12.172128 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.172026    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"
Oct 30 22:46:12.172128 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.172038    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" event={"ID":"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7","Type":"ContainerDied","Data":"62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405"}
Oct 30 22:46:12.172128 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.172092    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" event={"ID":"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7","Type":"ContainerDied","Data":"92d14e96ed32fa07232a8c8f19d73bba1484ccf78759ad62a3292bde8916f125"}
Oct 30 22:46:12.172128 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.172125    5060 scope.go:122] "RemoveContainer" containerID="62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405"
Oct 30 22:46:12.236444 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.236424    5060 scope.go:122] "RemoveContainer" containerID="621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5"
Oct 30 22:46:12.252054 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252030    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kserve-provision-location\") pod \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") "
Oct 30 22:46:12.252173 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252114    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tls-certs\") pod \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") "
Oct 30 22:46:12.252173 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252158    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-dshm\") pod \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") "
Oct 30 22:46:12.252283 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252193    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-home\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-home\") pod \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") "
Oct 30 22:46:12.252283 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252256    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tmp-dir\") pod \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") "
Oct 30 22:46:12.252376 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252293    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-cabundle-cert\") pod \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") "
Oct 30 22:46:12.252376 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252348    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kube-api-access-866kx\" (UniqueName: \"kubernetes.io/projected/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kube-api-access-866kx\") pod \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") "
Oct 30 22:46:12.252481 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252406    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-model-cache\") pod \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\" (UID: \"534c07e4-2a3c-4bcd-bc66-77f81d47b1b7\") "
Oct 30 22:46:12.252846 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252693    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-home" pod "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" (UID: "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:12.252846 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252804    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tmp-dir" pod "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" (UID: "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:12.253031 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.252943    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-model-cache" pod "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" (UID: "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:12.253090 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.253023    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-cabundle-cert" pod "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" (UID: "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:12.255065 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.255033    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tls-certs" pod "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" (UID: "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:12.255165 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.255080    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kube-api-access-866kx" pod "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" (UID: "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7"). InnerVolumeSpecName "kube-api-access-866kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:12.255618 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.255594    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-dshm" pod "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" (UID: "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:12.274361 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.274342    5060 scope.go:122] "RemoveContainer" containerID="62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405"
Oct 30 22:46:12.274848 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:46:12.274781    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405\": container with ID starting with 62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405 not found: ID does not exist" containerID="62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405"
Oct 30 22:46:12.274848 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.274818    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405"} err="failed to get container status \"62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405\": rpc error: code = NotFound desc = could not find container \"62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405\": container with ID starting with 62e8ca3b52cc9e82641b9b80219ccbc02b77f37d7edeac0331cc21140c2a9405 not found: ID does not exist"
Oct 30 22:46:12.274848 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.274845    5060 scope.go:122] "RemoveContainer" containerID="621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5"
Oct 30 22:46:12.275201 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:46:12.275145    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5\": container with ID starting with 621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5 not found: ID does not exist" containerID="621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5"
Oct 30 22:46:12.275201 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.275174    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5"} err="failed to get container status \"621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5\": rpc error: code = NotFound desc = could not find container \"621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5\": container with ID starting with 621fa0276c7f1cc6f0777b27cde573376a9e610937bd1f02a17256c9344cd8e5 not found: ID does not exist"
Oct 30 22:46:12.283550 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.282891    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kserve-provision-location" pod "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" (UID: "534c07e4-2a3c-4bcd-bc66-77f81d47b1b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:12.354264 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.354230    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.354264 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.354267    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.354492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.354282    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.354492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.354306    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.354492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.354328    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-866kx\" (UniqueName: \"kubernetes.io/projected/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kube-api-access-866kx\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.354492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.354355    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.354492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.354377    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.354492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.354392    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.413944 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.413913    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:46:12.500201 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.500166    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"]
Oct 30 22:46:12.506650 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.506619    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m"]
Oct 30 22:46:12.556527 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.556487    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-kserve-provision-location\") pod \"74a8a072-3630-4833-a205-7d61e5cfe292\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") "
Oct 30 22:46:12.556988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.556581    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-tmp-dir\") pod \"74a8a072-3630-4833-a205-7d61e5cfe292\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") "
Oct 30 22:46:12.556988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.556652    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a8a072-3630-4833-a205-7d61e5cfe292-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a8a072-3630-4833-a205-7d61e5cfe292-tls-certs\") pod \"74a8a072-3630-4833-a205-7d61e5cfe292\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") "
Oct 30 22:46:12.556988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.556747    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a8a072-3630-4833-a205-7d61e5cfe292-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74a8a072-3630-4833-a205-7d61e5cfe292-cabundle-cert\") pod \"74a8a072-3630-4833-a205-7d61e5cfe292\" (UID: \"74a8a072-3630-4833-a205-7d61e5cfe292\") "
Oct 30 22:46:12.557218 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.557114    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a8a072-3630-4833-a205-7d61e5cfe292-cabundle-cert" pod "74a8a072-3630-4833-a205-7d61e5cfe292" (UID: "74a8a072-3630-4833-a205-7d61e5cfe292"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:12.557342 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.557315    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-tmp-dir" pod "74a8a072-3630-4833-a205-7d61e5cfe292" (UID: "74a8a072-3630-4833-a205-7d61e5cfe292"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:12.558852 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.558833    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a8a072-3630-4833-a205-7d61e5cfe292-tls-certs" pod "74a8a072-3630-4833-a205-7d61e5cfe292" (UID: "74a8a072-3630-4833-a205-7d61e5cfe292"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:12.576863 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.576829    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-kserve-provision-location" pod "74a8a072-3630-4833-a205-7d61e5cfe292" (UID: "74a8a072-3630-4833-a205-7d61e5cfe292"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:12.658690 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.658606    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.658690 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.658655    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a8a072-3630-4833-a205-7d61e5cfe292-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.658690 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.658669    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a8a072-3630-4833-a205-7d61e5cfe292-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.658690 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:12.658680    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74a8a072-3630-4833-a205-7d61e5cfe292-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:13.032063 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.032016    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-79d7476f4b-v284m" podUID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded"
Oct 30 22:46:13.178199 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.178143    5060 generic.go:356] "Generic (PLEG): container finished" podID="74a8a072-3630-4833-a205-7d61e5cfe292" containerID="e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654" exitCode=0
Oct 30 22:46:13.178199 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.178199    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"
Oct 30 22:46:13.178457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.178224    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q" event={"ID":"74a8a072-3630-4833-a205-7d61e5cfe292","Type":"ContainerDied","Data":"e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654"}
Oct 30 22:46:13.178457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.178322    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q" event={"ID":"74a8a072-3630-4833-a205-7d61e5cfe292","Type":"ContainerDied","Data":"21aa8b4b719b865dd912f483c6b354522a21b9a4e2a7bc69ed7708729d806ded"}
Oct 30 22:46:13.178457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.178352    5060 scope.go:122] "RemoveContainer" containerID="e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654"
Oct 30 22:46:13.187787 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.187769    5060 scope.go:122] "RemoveContainer" containerID="8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a"
Oct 30 22:46:13.196696 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.196677    5060 scope.go:122] "RemoveContainer" containerID="e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654"
Oct 30 22:46:13.196945 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:46:13.196926    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654\": container with ID starting with e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654 not found: ID does not exist" containerID="e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654"
Oct 30 22:46:13.196994 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.196953    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654"} err="failed to get container status \"e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654\": rpc error: code = NotFound desc = could not find container \"e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654\": container with ID starting with e3a3b48a72db2bc31d9619da71ebe613115d782b10c0fcb975e758776365b654 not found: ID does not exist"
Oct 30 22:46:13.196994 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.196970    5060 scope.go:122] "RemoveContainer" containerID="8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a"
Oct 30 22:46:13.197185 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:46:13.197171    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a\": container with ID starting with 8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a not found: ID does not exist" containerID="8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a"
Oct 30 22:46:13.197225 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.197188    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a"} err="failed to get container status \"8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a\": rpc error: code = NotFound desc = could not find container \"8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a\": container with ID starting with 8d95f917ccae088dd6e8d13035163328a480695cd52b1035995074a10215982a not found: ID does not exist"
Oct 30 22:46:13.202362 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.202342    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"]
Oct 30 22:46:13.206760 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.206740    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6bbcf98bf7-5gs5q"]
Oct 30 22:46:13.290656 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.290577    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" path="/var/lib/kubelet/pods/534c07e4-2a3c-4bcd-bc66-77f81d47b1b7/volumes"
Oct 30 22:46:13.291226 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:13.291205    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a8a072-3630-4833-a205-7d61e5cfe292" path="/var/lib/kubelet/pods/74a8a072-3630-4833-a205-7d61e5cfe292/volumes"
Oct 30 22:46:14.547939 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:14.547898    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:46:24.547960 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:24.547846    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:46:34.547729 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:34.547676    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:46:34.775053 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:46:34.775008    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:46:44.547065 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:44.547017    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:46:54.547843 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:46:54.547802    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:47:04.547400 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:04.547358    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:47:04.775305 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:47:04.775253    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:47:08.290419 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.290369    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"]
Oct 30 22:47:08.291047 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.291023    5060 cpu_manager.go:414] "containerMap: removing container" podUID="74a8a072-3630-4833-a205-7d61e5cfe292" containerName="main"
Oct 30 22:47:08.291105 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.291053    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a8a072-3630-4833-a205-7d61e5cfe292" containerName="main"
Oct 30 22:47:08.291105 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.291071    5060 cpu_manager.go:414] "containerMap: removing container" podUID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" containerName="main"
Oct 30 22:47:08.291105 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.291080    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" containerName="main"
Oct 30 22:47:08.291215 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.291122    5060 cpu_manager.go:414] "containerMap: removing container" podUID="74a8a072-3630-4833-a205-7d61e5cfe292" containerName="storage-initializer"
Oct 30 22:47:08.291215 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.291130    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a8a072-3630-4833-a205-7d61e5cfe292" containerName="storage-initializer"
Oct 30 22:47:08.291215 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.291149    5060 cpu_manager.go:414] "containerMap: removing container" podUID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" containerName="storage-initializer"
Oct 30 22:47:08.291215 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.291156    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" containerName="storage-initializer"
Oct 30 22:47:08.291333 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.291268    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a8a072-3630-4833-a205-7d61e5cfe292" containerName="main"
Oct 30 22:47:08.291333 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.291282    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="534c07e4-2a3c-4bcd-bc66-77f81d47b1b7" containerName="main"
Oct 30 22:47:08.296266 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.296239    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.306852 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.306820    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"]
Oct 30 22:47:08.309524 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.309482    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Oct 30 22:47:08.309823 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.309680    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Oct 30 22:47:08.309823 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.309783    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-9s22j\""
Oct 30 22:47:08.359309 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.359273    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.359530 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.359338    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-home\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.359530 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.359421    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-model-cache\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.359530 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.359445    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834a794b-a388-4977-8235-9394ebcb42bf-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.359530 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.359466    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-dshm\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.359530 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.359500    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.359790 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.359603    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834a794b-a388-4977-8235-9394ebcb42bf-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.359790 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.359655    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv5gc\" (UniqueName: \"kubernetes.io/projected/834a794b-a388-4977-8235-9394ebcb42bf-kube-api-access-qv5gc\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.390794 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.390752    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Oct 30 22:47:08.391002 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.390902    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Oct 30 22:47:08.391002 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.390903    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:47:08.460543 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.460491    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.460543 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.460544    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834a794b-a388-4977-8235-9394ebcb42bf-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.460797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.460572    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv5gc\" (UniqueName: \"kubernetes.io/projected/834a794b-a388-4977-8235-9394ebcb42bf-kube-api-access-qv5gc\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.460797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.460611    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.460797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.460660    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-home\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.460797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.460709    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-model-cache\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.460797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.460729    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834a794b-a388-4977-8235-9394ebcb42bf-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.460797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.460764    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-dshm\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.461094 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.460931    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.461212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.461179    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-home\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.461212 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.461201    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-model-cache\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.461336 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.461300    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834a794b-a388-4977-8235-9394ebcb42bf-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.461427 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.461410    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.463048 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.463021    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-dshm\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.463744 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.463722    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834a794b-a388-4977-8235-9394ebcb42bf-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.476478 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.476447    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv5gc\" (UniqueName: \"kubernetes.io/projected/834a794b-a388-4977-8235-9394ebcb42bf-kube-api-access-qv5gc\") pod \"tokenizer-migration-path-test-kserve-8b767c9df-dq5z5\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.520337 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.520300    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"]
Oct 30 22:47:08.526886 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.526856    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.537585 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.537556    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"]
Oct 30 22:47:08.561903 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.561820    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.561903 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.561869    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.562137 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.562010    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.562137 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.562062    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.617934 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.617900    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:08.662733 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.662694    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.662929 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.662750    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.662929 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.662795    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.662929 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.662850    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.663218 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.663191    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.663300 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.663226    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.663584 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.663563    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.665525 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.665485    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.846649 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.846558    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:08.901349 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:08.901306    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"]
Oct 30 22:47:09.134924 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:09.134896    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"]
Oct 30 22:47:09.137810 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:47:09.137774    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6f57ad_dbd4_4f37_b708_cc1ba209a5b2.slice/crio-cf832b89a05b6d30e066c8db2a7622ad3c77bfc061e669c3f2198c0ae045a9fa WatchSource:0}: Error finding container cf832b89a05b6d30e066c8db2a7622ad3c77bfc061e669c3f2198c0ae045a9fa: Status 404 returned error can't find the container with id cf832b89a05b6d30e066c8db2a7622ad3c77bfc061e669c3f2198c0ae045a9fa
Oct 30 22:47:09.445462 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:09.445422    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" event={"ID":"834a794b-a388-4977-8235-9394ebcb42bf","Type":"ContainerStarted","Data":"2f7651cd5f300993986b3cfa6019b42a4105bcf67f6087e3fff5b10b290840ba"}
Oct 30 22:47:09.445973 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:09.445492    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" event={"ID":"834a794b-a388-4977-8235-9394ebcb42bf","Type":"ContainerStarted","Data":"27b98a7a7527964c3cc1e8db02191cd674729afbf706baacd9d715fd7d185f02"}
Oct 30 22:47:09.447301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:09.447269    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js" event={"ID":"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2","Type":"ContainerStarted","Data":"4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4"}
Oct 30 22:47:09.447436 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:09.447323    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js" event={"ID":"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2","Type":"ContainerStarted","Data":"cf832b89a05b6d30e066c8db2a7622ad3c77bfc061e669c3f2198c0ae045a9fa"}
Oct 30 22:47:12.487956 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:12.487919    5060 generic.go:356] "Generic (PLEG): container finished" podID="834a794b-a388-4977-8235-9394ebcb42bf" containerID="2f7651cd5f300993986b3cfa6019b42a4105bcf67f6087e3fff5b10b290840ba" exitCode=0
Oct 30 22:47:12.488408 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:12.487995    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" event={"ID":"834a794b-a388-4977-8235-9394ebcb42bf","Type":"ContainerDied","Data":"2f7651cd5f300993986b3cfa6019b42a4105bcf67f6087e3fff5b10b290840ba"}
Oct 30 22:47:12.489714 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:12.489693    5060 generic.go:356] "Generic (PLEG): container finished" podID="8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" containerID="4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4" exitCode=0
Oct 30 22:47:12.489828 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:12.489724    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js" event={"ID":"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2","Type":"ContainerDied","Data":"4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4"}
Oct 30 22:47:13.496881 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:13.496842    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js" event={"ID":"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2","Type":"ContainerStarted","Data":"813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff"}
Oct 30 22:47:13.497349 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:13.497107    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:13.499059 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:13.499031    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" event={"ID":"834a794b-a388-4977-8235-9394ebcb42bf","Type":"ContainerStarted","Data":"9daf9a8531c6de4c737c95b6899d5ee0f87d638d4ed856126dd979b7ebeb3f1c"}
Oct 30 22:47:13.521789 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:13.521731    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js" podStartSLOduration=2.840228589 podStartE2EDuration="5.521712394s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.681483805s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:13.519799881 +0000 UTC m=+1714.339537700" watchObservedRunningTime="2026-10-30 22:47:13.521712394 +0000 UTC m=+1714.341450216"
Oct 30 22:47:13.547080 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:13.547015    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" podStartSLOduration=2.848667679 podStartE2EDuration="5.546996932s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.698329253s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:13.54434872 +0000 UTC m=+1714.364086549" watchObservedRunningTime="2026-10-30 22:47:13.546996932 +0000 UTC m=+1714.366734749"
Oct 30 22:47:13.618874 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:13.618845    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:14.547271 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:14.547231    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:47:18.618435 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:18.618401    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:18.634204 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:18.634177    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:19.540821 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:19.540789    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:24.557006 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:24.556976    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:47:24.564391 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:24.564364    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:47:30.179991 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:30.179958    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"]
Oct 30 22:47:34.776525 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:47:34.776462    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:47:44.515147 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:44.515116    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:45.209122 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:45.209084    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" containerID="cri-o://74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e" gracePeriod=45
Oct 30 22:47:45.558101 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:45.558016    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Oct 30 22:47:45.955945 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:45.955917    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:47:46.035271 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035229    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-model-cache\") pod \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") "
Oct 30 22:47:46.035463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035286    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tmp-dir\") pod \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") "
Oct 30 22:47:46.035463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035314    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kube-api-access-zl5jx\" (UniqueName: \"kubernetes.io/projected/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kube-api-access-zl5jx\") pod \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") "
Oct 30 22:47:46.035463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035339    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-home\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-home\") pod \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") "
Oct 30 22:47:46.035463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035360    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-cabundle-cert\") pod \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") "
Oct 30 22:47:46.035463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035431    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kserve-provision-location\") pod \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") "
Oct 30 22:47:46.035463 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035463    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-dshm\") pod \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") "
Oct 30 22:47:46.035823 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035493    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tls-certs\") pod \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\" (UID: \"aa41b689-d2dc-401d-8f82-f87fb8e1ad10\") "
Oct 30 22:47:46.035823 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035490    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-model-cache" pod "aa41b689-d2dc-401d-8f82-f87fb8e1ad10" (UID: "aa41b689-d2dc-401d-8f82-f87fb8e1ad10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:46.035823 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035769    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.035980 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.035864    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-cabundle-cert" pod "aa41b689-d2dc-401d-8f82-f87fb8e1ad10" (UID: "aa41b689-d2dc-401d-8f82-f87fb8e1ad10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:46.036158 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.036124    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-home" pod "aa41b689-d2dc-401d-8f82-f87fb8e1ad10" (UID: "aa41b689-d2dc-401d-8f82-f87fb8e1ad10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:46.037776 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.037740    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-dshm" pod "aa41b689-d2dc-401d-8f82-f87fb8e1ad10" (UID: "aa41b689-d2dc-401d-8f82-f87fb8e1ad10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:46.037879 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.037843    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kube-api-access-zl5jx" pod "aa41b689-d2dc-401d-8f82-f87fb8e1ad10" (UID: "aa41b689-d2dc-401d-8f82-f87fb8e1ad10"). InnerVolumeSpecName "kube-api-access-zl5jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:46.038071 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.038048    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tls-certs" pod "aa41b689-d2dc-401d-8f82-f87fb8e1ad10" (UID: "aa41b689-d2dc-401d-8f82-f87fb8e1ad10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:46.045672 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.045647    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tmp-dir" pod "aa41b689-d2dc-401d-8f82-f87fb8e1ad10" (UID: "aa41b689-d2dc-401d-8f82-f87fb8e1ad10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:46.055746 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.055723    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kserve-provision-location" pod "aa41b689-d2dc-401d-8f82-f87fb8e1ad10" (UID: "aa41b689-d2dc-401d-8f82-f87fb8e1ad10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:46.136611 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.136529    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.136611 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.136557    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.136611 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.136566    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.136611 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.136576    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl5jx\" (UniqueName: \"kubernetes.io/projected/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kube-api-access-zl5jx\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.136611 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.136584    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.136611 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.136593    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.136611 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.136602    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa41b689-d2dc-401d-8f82-f87fb8e1ad10-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:46.256732 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.256697    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js_8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2/main/0.log"
Oct 30 22:47:46.505594 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.505555    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"]
Oct 30 22:47:46.515087 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:47:46.515054    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="9daf9a8531c6de4c737c95b6899d5ee0f87d638d4ed856126dd979b7ebeb3f1c" cmd=["/bin/sleep","15"]
Oct 30 22:47:46.515279 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.515099    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:47:46.515279 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:47:46.515112    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" podUID="834a794b-a388-4977-8235-9394ebcb42bf" containerName="main" containerID="cri-o://9daf9a8531c6de4c737c95b6899d5ee0f87d638d4ed856126dd979b7ebeb3f1c"
Oct 30 22:47:46.515279 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.515157    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" podUID="834a794b-a388-4977-8235-9394ebcb42bf" containerName="main" containerID="cri-o://9daf9a8531c6de4c737c95b6899d5ee0f87d638d4ed856126dd979b7ebeb3f1c" gracePeriod=60
Oct 30 22:47:46.524897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.524870    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"]
Oct 30 22:47:46.525098 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.525066    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js" podUID="8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" containerName="main" containerID="cri-o://813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff" gracePeriod=30
Oct 30 22:47:46.529830 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.529798    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" podUID="834a794b-a388-4977-8235-9394ebcb42bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 30 22:47:46.665073 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.665041    5060 generic.go:356] "Generic (PLEG): container finished" podID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerID="74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e" exitCode=0
Oct 30 22:47:46.665536 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.665116    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"
Oct 30 22:47:46.665536 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.665121    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" event={"ID":"aa41b689-d2dc-401d-8f82-f87fb8e1ad10","Type":"ContainerDied","Data":"74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e"}
Oct 30 22:47:46.665536 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.665182    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw" event={"ID":"aa41b689-d2dc-401d-8f82-f87fb8e1ad10","Type":"ContainerDied","Data":"6c578880dd5853fe0045ecc6a8ed06dc670295fc53c3b16e892c6a4da61a8409"}
Oct 30 22:47:46.665536 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.665208    5060 scope.go:122] "RemoveContainer" containerID="74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e"
Oct 30 22:47:46.667245 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.667223    5060 generic.go:356] "Generic (PLEG): container finished" podID="834a794b-a388-4977-8235-9394ebcb42bf" containerID="9daf9a8531c6de4c737c95b6899d5ee0f87d638d4ed856126dd979b7ebeb3f1c" exitCode=0
Oct 30 22:47:46.667368 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.667261    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" event={"ID":"834a794b-a388-4977-8235-9394ebcb42bf","Type":"ContainerDied","Data":"9daf9a8531c6de4c737c95b6899d5ee0f87d638d4ed856126dd979b7ebeb3f1c"}
Oct 30 22:47:46.679236 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.679219    5060 scope.go:122] "RemoveContainer" containerID="eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff"
Oct 30 22:47:46.686928 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.686912    5060 scope.go:122] "RemoveContainer" containerID="74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e"
Oct 30 22:47:46.687206 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:47:46.687185    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e\": container with ID starting with 74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e not found: ID does not exist" containerID="74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e"
Oct 30 22:47:46.687262 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.687214    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e"} err="failed to get container status \"74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e\": rpc error: code = NotFound desc = could not find container \"74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e\": container with ID starting with 74a7d00f1066a68d183b664d169b117265f3accd2d6d3a96879fdd5f2c1b3a8e not found: ID does not exist"
Oct 30 22:47:46.687262 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.687232    5060 scope.go:122] "RemoveContainer" containerID="eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff"
Oct 30 22:47:46.687462 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:47:46.687443    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff\": container with ID starting with eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff not found: ID does not exist" containerID="eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff"
Oct 30 22:47:46.687531 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.687472    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff"} err="failed to get container status \"eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff\": rpc error: code = NotFound desc = could not find container \"eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff\": container with ID starting with eeef6d0db4d81a0f65f9d0304cefd524e65c87b7a1160728888fe776522812ff not found: ID does not exist"
Oct 30 22:47:46.693804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.693781    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"]
Oct 30 22:47:46.698758 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:46.698739    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65ddcfbf9c-wjgsw"]
Oct 30 22:47:47.179028 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.179002    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:47.242794 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.242763    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-cabundle-cert\") pod \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") "
Oct 30 22:47:47.242957 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.242834    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tmp-dir\") pod \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") "
Oct 30 22:47:47.242957 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.242871    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tls-certs\") pod \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") "
Oct 30 22:47:47.242957 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.242892    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-kserve-provision-location\") pod \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\" (UID: \"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2\") "
Oct 30 22:47:47.243206 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.243179    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-cabundle-cert" pod "8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" (UID: "8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:47.243318 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.243295    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tmp-dir" pod "8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" (UID: "8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.246583 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.246556    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tls-certs" pod "8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" (UID: "8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:47.270734 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.270701    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-kserve-provision-location" pod "8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" (UID: "8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.281750 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.281726    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:47.288881 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.288857    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" path="/var/lib/kubelet/pods/aa41b689-d2dc-401d-8f82-f87fb8e1ad10/volumes"
Oct 30 22:47:47.343672 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.343643    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-model-cache\") pod \"834a794b-a388-4977-8235-9394ebcb42bf\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") "
Oct 30 22:47:47.343828 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.343677    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/834a794b-a388-4977-8235-9394ebcb42bf-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834a794b-a388-4977-8235-9394ebcb42bf-cabundle-cert\") pod \"834a794b-a388-4977-8235-9394ebcb42bf\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") "
Oct 30 22:47:47.343828 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.343703    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/834a794b-a388-4977-8235-9394ebcb42bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/834a794b-a388-4977-8235-9394ebcb42bf-tls-certs\") pod \"834a794b-a388-4977-8235-9394ebcb42bf\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") "
Oct 30 22:47:47.343828 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.343731    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-dshm\") pod \"834a794b-a388-4977-8235-9394ebcb42bf\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") "
Oct 30 22:47:47.343828 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.343760    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-kserve-provision-location\") pod \"834a794b-a388-4977-8235-9394ebcb42bf\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") "
Oct 30 22:47:47.343828 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.343793    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/834a794b-a388-4977-8235-9394ebcb42bf-kube-api-access-qv5gc\" (UniqueName: \"kubernetes.io/projected/834a794b-a388-4977-8235-9394ebcb42bf-kube-api-access-qv5gc\") pod \"834a794b-a388-4977-8235-9394ebcb42bf\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") "
Oct 30 22:47:47.344137 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.343843    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-home\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-home\") pod \"834a794b-a388-4977-8235-9394ebcb42bf\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") "
Oct 30 22:47:47.344137 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.343893    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-tmp-dir\") pod \"834a794b-a388-4977-8235-9394ebcb42bf\" (UID: \"834a794b-a388-4977-8235-9394ebcb42bf\") "
Oct 30 22:47:47.344137 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.343923    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-model-cache" pod "834a794b-a388-4977-8235-9394ebcb42bf" (UID: "834a794b-a388-4977-8235-9394ebcb42bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.344137 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.344042    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834a794b-a388-4977-8235-9394ebcb42bf-cabundle-cert" pod "834a794b-a388-4977-8235-9394ebcb42bf" (UID: "834a794b-a388-4977-8235-9394ebcb42bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:47.344341 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.344197    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-home" pod "834a794b-a388-4977-8235-9394ebcb42bf" (UID: "834a794b-a388-4977-8235-9394ebcb42bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.344341 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.344208    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-tmp-dir" pod "834a794b-a388-4977-8235-9394ebcb42bf" (UID: "834a794b-a388-4977-8235-9394ebcb42bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.344341 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.344223    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.344341 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.344242    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/834a794b-a388-4977-8235-9394ebcb42bf-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.344341 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.344255    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.344341 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.344263    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.344341 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.344271    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.344341 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.344280    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.345964 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.345937    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-dshm" pod "834a794b-a388-4977-8235-9394ebcb42bf" (UID: "834a794b-a388-4977-8235-9394ebcb42bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.345964 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.345948    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834a794b-a388-4977-8235-9394ebcb42bf-tls-certs" pod "834a794b-a388-4977-8235-9394ebcb42bf" (UID: "834a794b-a388-4977-8235-9394ebcb42bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:47.345964 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.345951    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834a794b-a388-4977-8235-9394ebcb42bf-kube-api-access-qv5gc" pod "834a794b-a388-4977-8235-9394ebcb42bf" (UID: "834a794b-a388-4977-8235-9394ebcb42bf"). InnerVolumeSpecName "kube-api-access-qv5gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:47.365222 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.365199    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-kserve-provision-location" pod "834a794b-a388-4977-8235-9394ebcb42bf" (UID: "834a794b-a388-4977-8235-9394ebcb42bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:47.445245 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.445214    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.445245 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.445240    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.445245 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.445249    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834a794b-a388-4977-8235-9394ebcb42bf-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.445434 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.445257    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.445434 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.445270    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/834a794b-a388-4977-8235-9394ebcb42bf-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.445434 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.445282    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv5gc\" (UniqueName: \"kubernetes.io/projected/834a794b-a388-4977-8235-9394ebcb42bf-kube-api-access-qv5gc\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:47.673104 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.673018    5060 generic.go:356] "Generic (PLEG): container finished" podID="8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" containerID="813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff" exitCode=0
Oct 30 22:47:47.673104 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.673078    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"
Oct 30 22:47:47.673646 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.673100    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js" event={"ID":"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2","Type":"ContainerDied","Data":"813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff"}
Oct 30 22:47:47.673646 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.673147    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js" event={"ID":"8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2","Type":"ContainerDied","Data":"cf832b89a05b6d30e066c8db2a7622ad3c77bfc061e669c3f2198c0ae045a9fa"}
Oct 30 22:47:47.673646 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.673170    5060 scope.go:122] "RemoveContainer" containerID="813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff"
Oct 30 22:47:47.675777 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.675757    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5" event={"ID":"834a794b-a388-4977-8235-9394ebcb42bf","Type":"ContainerDied","Data":"27b98a7a7527964c3cc1e8db02191cd674729afbf706baacd9d715fd7d185f02"}
Oct 30 22:47:47.675882 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.675776    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"
Oct 30 22:47:47.682017 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.682000    5060 scope.go:122] "RemoveContainer" containerID="4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4"
Oct 30 22:47:47.689589 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.689570    5060 scope.go:122] "RemoveContainer" containerID="813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff"
Oct 30 22:47:47.689844 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:47:47.689824    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff\": container with ID starting with 813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff not found: ID does not exist" containerID="813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff"
Oct 30 22:47:47.689882 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.689852    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff"} err="failed to get container status \"813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff\": rpc error: code = NotFound desc = could not find container \"813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff\": container with ID starting with 813aeb4cdbcfc459bc4b6f54b998509ff14c80b65589c6cbbb96fdfa1f20e6ff not found: ID does not exist"
Oct 30 22:47:47.689882 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.689869    5060 scope.go:122] "RemoveContainer" containerID="4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4"
Oct 30 22:47:47.690096 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:47:47.690078    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4\": container with ID starting with 4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4 not found: ID does not exist" containerID="4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4"
Oct 30 22:47:47.690129 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.690104    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4"} err="failed to get container status \"4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4\": rpc error: code = NotFound desc = could not find container \"4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4\": container with ID starting with 4e474df8679c962d7a1506fbad529a1886fb4315971388a0dbd7b3c41fbe1df4 not found: ID does not exist"
Oct 30 22:47:47.690129 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.690120    5060 scope.go:122] "RemoveContainer" containerID="9daf9a8531c6de4c737c95b6899d5ee0f87d638d4ed856126dd979b7ebeb3f1c"
Oct 30 22:47:47.695349 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.695331    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"]
Oct 30 22:47:47.698243 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.698222    5060 scope.go:122] "RemoveContainer" containerID="2f7651cd5f300993986b3cfa6019b42a4105bcf67f6087e3fff5b10b290840ba"
Oct 30 22:47:47.700322 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.700302    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-5cd7c9f67f-9m9js"]
Oct 30 22:47:47.717378 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.717357    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"]
Oct 30 22:47:47.723004 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:47.722983    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8b767c9df-dq5z5"]
Oct 30 22:47:49.289593 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:49.289564    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="834a794b-a388-4977-8235-9394ebcb42bf" path="/var/lib/kubelet/pods/834a794b-a388-4977-8235-9394ebcb42bf/volumes"
Oct 30 22:47:49.289987 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:47:49.289974    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" path="/var/lib/kubelet/pods/8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2/volumes"
Oct 30 22:48:04.775689 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:48:04.775652    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:48:34.774834 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:48:34.774788    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:48:38.011015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.010984    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"]
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011364    5060 cpu_manager.go:414] "containerMap: removing container" podUID="834a794b-a388-4977-8235-9394ebcb42bf" containerName="main"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011378    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="834a794b-a388-4977-8235-9394ebcb42bf" containerName="main"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011393    5060 cpu_manager.go:414] "containerMap: removing container" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011399    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011415    5060 cpu_manager.go:414] "containerMap: removing container" podUID="834a794b-a388-4977-8235-9394ebcb42bf" containerName="storage-initializer"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011420    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="834a794b-a388-4977-8235-9394ebcb42bf" containerName="storage-initializer"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011427    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" containerName="main"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011432    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" containerName="main"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011443    5060 cpu_manager.go:414] "containerMap: removing container" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="storage-initializer"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011450    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="storage-initializer"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011466    5060 cpu_manager.go:414] "containerMap: removing container" podUID="8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" containerName="storage-initializer"
Oct 30 22:48:38.011477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011471    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" containerName="storage-initializer"
Oct 30 22:48:38.011852 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011554    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="834a794b-a388-4977-8235-9394ebcb42bf" containerName="main"
Oct 30 22:48:38.011852 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011562    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa41b689-d2dc-401d-8f82-f87fb8e1ad10" containerName="main"
Oct 30 22:48:38.011852 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.011570    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6f57ad-dbd4-4f37-b708-cc1ba209a5b2" containerName="main"
Oct 30 22:48:38.014702 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.014682    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.018983 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.018959    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Oct 30 22:48:38.019218 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.019197    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Oct 30 22:48:38.019308 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.019223    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-5f72j\""
Oct 30 22:48:38.020269 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.020250    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Oct 30 22:48:38.020405 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.020312    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:48:38.020467 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.020313    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Oct 30 22:48:38.028328 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.028306    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"]
Oct 30 22:48:38.091575 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.091542    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-dshm\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.091575 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.091575    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c67e394-3cfa-4438-ada6-62ae4514184f-tls-certs\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.091775 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.091607    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2fss\" (UniqueName: \"kubernetes.io/projected/6c67e394-3cfa-4438-ada6-62ae4514184f-kube-api-access-f2fss\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.091775 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.091743    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-model-cache\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.091843 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.091782    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.091843 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.091809    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c67e394-3cfa-4438-ada6-62ae4514184f-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.091843 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.091830    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-home\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.091936 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.091857    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.192762 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.192722    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-dshm\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.192762 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.192766    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c67e394-3cfa-4438-ada6-62ae4514184f-tls-certs\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193024 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.192799    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2fss\" (UniqueName: \"kubernetes.io/projected/6c67e394-3cfa-4438-ada6-62ae4514184f-kube-api-access-f2fss\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193024 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.192845    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-model-cache\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193024 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.192889    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193024 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.192927    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c67e394-3cfa-4438-ada6-62ae4514184f-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193024 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.192949    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-home\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193024 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.192966    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193403 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.193366    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193552 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.193383    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-model-cache\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193552 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.193474    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193686 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.193587    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-home\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.193742 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.193731    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c67e394-3cfa-4438-ada6-62ae4514184f-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.195058 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.195038    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-dshm\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.195414 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.195398    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c67e394-3cfa-4438-ada6-62ae4514184f-tls-certs\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.207094 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.207072    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2fss\" (UniqueName: \"kubernetes.io/projected/6c67e394-3cfa-4438-ada6-62ae4514184f-kube-api-access-f2fss\") pod \"tokenizer-template-args-test-kserve-6784ddf685-pvg9n\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.317305 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.317227    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"]
Oct 30 22:48:38.321195 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.321169    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.330066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.330040    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:38.338645 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.338615    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"]
Oct 30 22:48:38.394542 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.394201    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.394542 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.394274    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.394542 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.394304    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.394542 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.394350    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.495597 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.495563    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.495787 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.495617    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.495787 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.495645    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.495787 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.495689    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.495993 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.495972    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.496064 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.496043    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.496279 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.496263    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.498136 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.498112    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.610760 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.610699    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"]
Oct 30 22:48:38.613067 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:48:38.613040    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c67e394_3cfa_4438_ada6_62ae4514184f.slice/crio-dfba73c382629bdef92f771ac9d2ecb43e4f37f859b412d46cd3855ccebb4acd WatchSource:0}: Error finding container dfba73c382629bdef92f771ac9d2ecb43e4f37f859b412d46cd3855ccebb4acd: Status 404 returned error can't find the container with id dfba73c382629bdef92f771ac9d2ecb43e4f37f859b412d46cd3855ccebb4acd
Oct 30 22:48:38.645420 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.645398    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:38.910813 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.910729    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n" event={"ID":"6c67e394-3cfa-4438-ada6-62ae4514184f","Type":"ContainerStarted","Data":"7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2"}
Oct 30 22:48:38.910813 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.910800    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n" event={"ID":"6c67e394-3cfa-4438-ada6-62ae4514184f","Type":"ContainerStarted","Data":"dfba73c382629bdef92f771ac9d2ecb43e4f37f859b412d46cd3855ccebb4acd"}
Oct 30 22:48:38.927825 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:38.927800    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"]
Oct 30 22:48:38.930686 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:48:38.930659    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c07f52_7c2e_4c3e_a578_344ed9d2db64.slice/crio-a681d0461d17f3a070c9c8d80ed34f5887a2a987843befbed39489ef2db2ff23 WatchSource:0}: Error finding container a681d0461d17f3a070c9c8d80ed34f5887a2a987843befbed39489ef2db2ff23: Status 404 returned error can't find the container with id a681d0461d17f3a070c9c8d80ed34f5887a2a987843befbed39489ef2db2ff23
Oct 30 22:48:39.507988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:39.507699    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:48:39.507988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:39.507923    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:48:39.934584 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:39.927711    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm" event={"ID":"d6c07f52-7c2e-4c3e-a578-344ed9d2db64","Type":"ContainerStarted","Data":"1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363"}
Oct 30 22:48:39.934584 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:39.927771    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm" event={"ID":"d6c07f52-7c2e-4c3e-a578-344ed9d2db64","Type":"ContainerStarted","Data":"a681d0461d17f3a070c9c8d80ed34f5887a2a987843befbed39489ef2db2ff23"}
Oct 30 22:48:42.974534 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:42.973832    5060 generic.go:356] "Generic (PLEG): container finished" podID="6c67e394-3cfa-4438-ada6-62ae4514184f" containerID="7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2" exitCode=0
Oct 30 22:48:42.974534 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:42.973950    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n" event={"ID":"6c67e394-3cfa-4438-ada6-62ae4514184f","Type":"ContainerDied","Data":"7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2"}
Oct 30 22:48:42.993640 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:42.976556    5060 generic.go:356] "Generic (PLEG): container finished" podID="d6c07f52-7c2e-4c3e-a578-344ed9d2db64" containerID="1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363" exitCode=0
Oct 30 22:48:42.993640 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:42.976713    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm" event={"ID":"d6c07f52-7c2e-4c3e-a578-344ed9d2db64","Type":"ContainerDied","Data":"1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363"}
Oct 30 22:48:44.045549 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:44.035285    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n" event={"ID":"6c67e394-3cfa-4438-ada6-62ae4514184f","Type":"ContainerStarted","Data":"5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501"}
Oct 30 22:48:44.045549 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:44.043427    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm" event={"ID":"d6c07f52-7c2e-4c3e-a578-344ed9d2db64","Type":"ContainerStarted","Data":"713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887"}
Oct 30 22:48:44.068309 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:44.068247    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:48:44.076543 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:44.074085    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm" podStartSLOduration=2.160525211 podStartE2EDuration="6.074067131s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.91354192s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:44.071993718 +0000 UTC m=+1804.891731577" watchObservedRunningTime="2026-10-30 22:48:44.074067131 +0000 UTC m=+1804.893804951"
Oct 30 22:48:44.333598 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:44.331614    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:48.330436 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:48.330402    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:48.342496 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:48.342469    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:48:48.367628 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:48.367581    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n" podStartSLOduration=7.465126896 podStartE2EDuration="11.367567s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.902440104s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:44.100825448 +0000 UTC m=+1804.920563269" watchObservedRunningTime="2026-10-30 22:48:48.367567 +0000 UTC m=+1809.187304813"
Oct 30 22:48:49.083594 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:48:49.083555    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:49:04.776158 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:49:04.776112    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:49:15.065464 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:15.065430    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:49:16.958935 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:16.958899    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm_d6c07f52-7c2e-4c3e-a578-344ed9d2db64/main/0.log"
Oct 30 22:49:17.208092 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.208060    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"]
Oct 30 22:49:17.218334 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:49:17.218256    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501" cmd=["/bin/sleep","15"]
Oct 30 22:49:17.218334 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.218304    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:49:17.218334 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:49:17.218317    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n" podUID="6c67e394-3cfa-4438-ada6-62ae4514184f" containerName="main" containerID="cri-o://5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501"
Oct 30 22:49:17.218607 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.218351    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n" podUID="6c67e394-3cfa-4438-ada6-62ae4514184f" containerName="main" containerID="cri-o://5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501" gracePeriod=60
Oct 30 22:49:17.226106 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.226075    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"]
Oct 30 22:49:17.226343 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.226318    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm" podUID="d6c07f52-7c2e-4c3e-a578-344ed9d2db64" containerName="main" containerID="cri-o://713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887" gracePeriod=30
Oct 30 22:49:17.578202 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.578168    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:49:17.670790 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.670762    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-kserve-provision-location\") pod \"6c67e394-3cfa-4438-ada6-62ae4514184f\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") "
Oct 30 22:49:17.670932 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.670813    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c67e394-3cfa-4438-ada6-62ae4514184f-kube-api-access-f2fss\" (UniqueName: \"kubernetes.io/projected/6c67e394-3cfa-4438-ada6-62ae4514184f-kube-api-access-f2fss\") pod \"6c67e394-3cfa-4438-ada6-62ae4514184f\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") "
Oct 30 22:49:17.670932 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.670848    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-dshm\") pod \"6c67e394-3cfa-4438-ada6-62ae4514184f\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") "
Oct 30 22:49:17.670932 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.670881    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c67e394-3cfa-4438-ada6-62ae4514184f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c67e394-3cfa-4438-ada6-62ae4514184f-tls-certs\") pod \"6c67e394-3cfa-4438-ada6-62ae4514184f\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") "
Oct 30 22:49:17.670932 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.670908    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-home\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-home\") pod \"6c67e394-3cfa-4438-ada6-62ae4514184f\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") "
Oct 30 22:49:17.671096 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.670944    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c67e394-3cfa-4438-ada6-62ae4514184f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c67e394-3cfa-4438-ada6-62ae4514184f-cabundle-cert\") pod \"6c67e394-3cfa-4438-ada6-62ae4514184f\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") "
Oct 30 22:49:17.671096 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.671031    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-tmp-dir\") pod \"6c67e394-3cfa-4438-ada6-62ae4514184f\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") "
Oct 30 22:49:17.671204 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.671098    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-model-cache\") pod \"6c67e394-3cfa-4438-ada6-62ae4514184f\" (UID: \"6c67e394-3cfa-4438-ada6-62ae4514184f\") "
Oct 30 22:49:17.671469 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.671435    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-tmp-dir" pod "6c67e394-3cfa-4438-ada6-62ae4514184f" (UID: "6c67e394-3cfa-4438-ada6-62ae4514184f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:17.671646 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.671470    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c67e394-3cfa-4438-ada6-62ae4514184f-cabundle-cert" pod "6c67e394-3cfa-4438-ada6-62ae4514184f" (UID: "6c67e394-3cfa-4438-ada6-62ae4514184f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:17.671646 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.671562    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-home" pod "6c67e394-3cfa-4438-ada6-62ae4514184f" (UID: "6c67e394-3cfa-4438-ada6-62ae4514184f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:17.671821 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.671577    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-model-cache" pod "6c67e394-3cfa-4438-ada6-62ae4514184f" (UID: "6c67e394-3cfa-4438-ada6-62ae4514184f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:17.673160 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.673136    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c67e394-3cfa-4438-ada6-62ae4514184f-tls-certs" pod "6c67e394-3cfa-4438-ada6-62ae4514184f" (UID: "6c67e394-3cfa-4438-ada6-62ae4514184f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:17.673390 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.673369    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-dshm" pod "6c67e394-3cfa-4438-ada6-62ae4514184f" (UID: "6c67e394-3cfa-4438-ada6-62ae4514184f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:17.673476 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.673390    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c67e394-3cfa-4438-ada6-62ae4514184f-kube-api-access-f2fss" pod "6c67e394-3cfa-4438-ada6-62ae4514184f" (UID: "6c67e394-3cfa-4438-ada6-62ae4514184f"). InnerVolumeSpecName "kube-api-access-f2fss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:17.693679 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.693650    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-kserve-provision-location" pod "6c67e394-3cfa-4438-ada6-62ae4514184f" (UID: "6c67e394-3cfa-4438-ada6-62ae4514184f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:17.772848 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.772808    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2fss\" (UniqueName: \"kubernetes.io/projected/6c67e394-3cfa-4438-ada6-62ae4514184f-kube-api-access-f2fss\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.772848 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.772845    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.772848 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.772855    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c67e394-3cfa-4438-ada6-62ae4514184f-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.773133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.772863    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.773133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.772873    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c67e394-3cfa-4438-ada6-62ae4514184f-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.773133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.772881    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.773133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.772893    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.773133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.772906    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c67e394-3cfa-4438-ada6-62ae4514184f-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:17.871000 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.870974    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:49:17.975401 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.975356    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tmp-dir\") pod \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") "
Oct 30 22:49:17.975892 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.975463    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tls-certs\") pod \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") "
Oct 30 22:49:17.975892 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.975486    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-kserve-provision-location\") pod \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") "
Oct 30 22:49:17.975892 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.975525    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-cabundle-cert\") pod \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\" (UID: \"d6c07f52-7c2e-4c3e-a578-344ed9d2db64\") "
Oct 30 22:49:17.976016 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.975902    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tmp-dir" pod "d6c07f52-7c2e-4c3e-a578-344ed9d2db64" (UID: "d6c07f52-7c2e-4c3e-a578-344ed9d2db64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:17.976016 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.975916    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-cabundle-cert" pod "d6c07f52-7c2e-4c3e-a578-344ed9d2db64" (UID: "d6c07f52-7c2e-4c3e-a578-344ed9d2db64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:17.977545 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.977522    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tls-certs" pod "d6c07f52-7c2e-4c3e-a578-344ed9d2db64" (UID: "d6c07f52-7c2e-4c3e-a578-344ed9d2db64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:17.999659 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:17.999596    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-kserve-provision-location" pod "d6c07f52-7c2e-4c3e-a578-344ed9d2db64" (UID: "d6c07f52-7c2e-4c3e-a578-344ed9d2db64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:18.077066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.077019    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:18.077066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.077062    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:18.077066 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.077073    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:18.077278 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.077084    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6c07f52-7c2e-4c3e-a578-344ed9d2db64-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:18.223802 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.223769    5060 generic.go:356] "Generic (PLEG): container finished" podID="6c67e394-3cfa-4438-ada6-62ae4514184f" containerID="5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501" exitCode=0
Oct 30 22:49:18.223988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.223824    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"
Oct 30 22:49:18.223988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.223841    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n" event={"ID":"6c67e394-3cfa-4438-ada6-62ae4514184f","Type":"ContainerDied","Data":"5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501"}
Oct 30 22:49:18.223988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.223883    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n" event={"ID":"6c67e394-3cfa-4438-ada6-62ae4514184f","Type":"ContainerDied","Data":"dfba73c382629bdef92f771ac9d2ecb43e4f37f859b412d46cd3855ccebb4acd"}
Oct 30 22:49:18.223988 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.223906    5060 scope.go:122] "RemoveContainer" containerID="5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501"
Oct 30 22:49:18.225767 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.225735    5060 generic.go:356] "Generic (PLEG): container finished" podID="d6c07f52-7c2e-4c3e-a578-344ed9d2db64" containerID="713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887" exitCode=0
Oct 30 22:49:18.225888 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.225781    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm" event={"ID":"d6c07f52-7c2e-4c3e-a578-344ed9d2db64","Type":"ContainerDied","Data":"713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887"}
Oct 30 22:49:18.225888 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.225835    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm" event={"ID":"d6c07f52-7c2e-4c3e-a578-344ed9d2db64","Type":"ContainerDied","Data":"a681d0461d17f3a070c9c8d80ed34f5887a2a987843befbed39489ef2db2ff23"}
Oct 30 22:49:18.225888 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.225790    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"
Oct 30 22:49:18.232951 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.232928    5060 scope.go:122] "RemoveContainer" containerID="7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2"
Oct 30 22:49:18.240795 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.240778    5060 scope.go:122] "RemoveContainer" containerID="5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501"
Oct 30 22:49:18.241049 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:49:18.241033    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501\": container with ID starting with 5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501 not found: ID does not exist" containerID="5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501"
Oct 30 22:49:18.241120 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.241061    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501"} err="failed to get container status \"5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501\": rpc error: code = NotFound desc = could not find container \"5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501\": container with ID starting with 5743a244a2173f1263466998a46463f5610aa2a09b0d7e06bb1b4f7a02b5f501 not found: ID does not exist"
Oct 30 22:49:18.241120 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.241086    5060 scope.go:122] "RemoveContainer" containerID="7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2"
Oct 30 22:49:18.241319 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:49:18.241301    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2\": container with ID starting with 7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2 not found: ID does not exist" containerID="7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2"
Oct 30 22:49:18.241359 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.241326    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2"} err="failed to get container status \"7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2\": rpc error: code = NotFound desc = could not find container \"7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2\": container with ID starting with 7a5d8b0654946b344b890741776e34cd168597df72aa017e9e7ce36683196ce2 not found: ID does not exist"
Oct 30 22:49:18.241359 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.241343    5060 scope.go:122] "RemoveContainer" containerID="713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887"
Oct 30 22:49:18.248735 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.248681    5060 scope.go:122] "RemoveContainer" containerID="1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363"
Oct 30 22:49:18.251133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.251070    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"]
Oct 30 22:49:18.255111 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.255089    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6784ddf685-pvg9n"]
Oct 30 22:49:18.256640 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.256622    5060 scope.go:122] "RemoveContainer" containerID="713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887"
Oct 30 22:49:18.256915 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:49:18.256893    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887\": container with ID starting with 713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887 not found: ID does not exist" containerID="713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887"
Oct 30 22:49:18.257001 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.256924    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887"} err="failed to get container status \"713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887\": rpc error: code = NotFound desc = could not find container \"713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887\": container with ID starting with 713590c139612c2042c34c77b953d772882b3dad628dd5242c2b3c2b08f22887 not found: ID does not exist"
Oct 30 22:49:18.257001 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.256946    5060 scope.go:122] "RemoveContainer" containerID="1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363"
Oct 30 22:49:18.257183 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:49:18.257167    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363\": container with ID starting with 1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363 not found: ID does not exist" containerID="1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363"
Oct 30 22:49:18.257219 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.257189    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363"} err="failed to get container status \"1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363\": rpc error: code = NotFound desc = could not find container \"1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363\": container with ID starting with 1efdd9e0386815f2917220f81925e06c8fc46188049552ba20dcfb887ec42363 not found: ID does not exist"
Oct 30 22:49:18.270587 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.270563    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"]
Oct 30 22:49:18.277534 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:18.277494    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b5cc5746b-qqhpm"]
Oct 30 22:49:19.289605 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:19.289573    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c67e394-3cfa-4438-ada6-62ae4514184f" path="/var/lib/kubelet/pods/6c67e394-3cfa-4438-ada6-62ae4514184f/volumes"
Oct 30 22:49:19.290006 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:49:19.289992    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c07f52-7c2e-4c3e-a578-344ed9d2db64" path="/var/lib/kubelet/pods/d6c07f52-7c2e-4c3e-a578-344ed9d2db64/volumes"
Oct 30 22:49:34.775114 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:49:34.775080    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:50:04.778099 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:50:04.778053    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:50:09.529995 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.529953    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"]
Oct 30 22:50:09.530436 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.530335    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6c67e394-3cfa-4438-ada6-62ae4514184f" containerName="storage-initializer"
Oct 30 22:50:09.530436 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.530347    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c67e394-3cfa-4438-ada6-62ae4514184f" containerName="storage-initializer"
Oct 30 22:50:09.530436 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.530358    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d6c07f52-7c2e-4c3e-a578-344ed9d2db64" containerName="storage-initializer"
Oct 30 22:50:09.530436 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.530363    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c07f52-7c2e-4c3e-a578-344ed9d2db64" containerName="storage-initializer"
Oct 30 22:50:09.530436 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.530375    5060 cpu_manager.go:414] "containerMap: removing container" podUID="6c67e394-3cfa-4438-ada6-62ae4514184f" containerName="main"
Oct 30 22:50:09.530436 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.530381    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c67e394-3cfa-4438-ada6-62ae4514184f" containerName="main"
Oct 30 22:50:09.530436 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.530400    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d6c07f52-7c2e-4c3e-a578-344ed9d2db64" containerName="main"
Oct 30 22:50:09.530436 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.530404    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c07f52-7c2e-4c3e-a578-344ed9d2db64" containerName="main"
Oct 30 22:50:09.530719 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.530460    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c67e394-3cfa-4438-ada6-62ae4514184f" containerName="main"
Oct 30 22:50:09.530719 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.530471    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c07f52-7c2e-4c3e-a578-344ed9d2db64" containerName="main"
Oct 30 22:50:09.533473 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.533457    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.538811 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.538780    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Oct 30 22:50:09.538951 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.538850    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-8cj2k\""
Oct 30 22:50:09.539192 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.539178    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Oct 30 22:50:09.539254 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.539202    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Oct 30 22:50:09.546373 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.546345    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"]
Oct 30 22:50:09.630320 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.630279    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.630320 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.630317    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.630596 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.630338    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jctk\" (UniqueName: \"kubernetes.io/projected/f40a398d-2577-4ecc-9c4c-e1ff431a6144-kube-api-access-8jctk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.630596 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.630485    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.630596 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.630556    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.630596 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.630587    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.731883 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.731841    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.731883 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.731881    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.732128 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.731902    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jctk\" (UniqueName: \"kubernetes.io/projected/f40a398d-2577-4ecc-9c4c-e1ff431a6144-kube-api-access-8jctk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.732128 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.731951    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.732128 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.731975    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.732128 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.732006    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.732331 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.732267    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.732442 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.732419    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.732530 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.732424    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.734138 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.734120    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.734447 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.734428    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.745121 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.745088    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jctk\" (UniqueName: \"kubernetes.io/projected/f40a398d-2577-4ecc-9c4c-e1ff431a6144-kube-api-access-8jctk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:09.846973 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:09.846877    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:10.124518 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:10.124482    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"]
Oct 30 22:50:10.126534 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:50:10.126497    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40a398d_2577_4ecc_9c4c_e1ff431a6144.slice/crio-7b40e0b7d77e0a5dbc9c739e26a75e20aa839ffcffaaa4e069872b8c44e93927 WatchSource:0}: Error finding container 7b40e0b7d77e0a5dbc9c739e26a75e20aa839ffcffaaa4e069872b8c44e93927: Status 404 returned error can't find the container with id 7b40e0b7d77e0a5dbc9c739e26a75e20aa839ffcffaaa4e069872b8c44e93927
Oct 30 22:50:10.460490 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:10.460448    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg" event={"ID":"f40a398d-2577-4ecc-9c4c-e1ff431a6144","Type":"ContainerStarted","Data":"ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61"}
Oct 30 22:50:10.460692 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:10.460532    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg" event={"ID":"f40a398d-2577-4ecc-9c4c-e1ff431a6144","Type":"ContainerStarted","Data":"7b40e0b7d77e0a5dbc9c739e26a75e20aa839ffcffaaa4e069872b8c44e93927"}
Oct 30 22:50:10.486868 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:10.486823    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg" podStartSLOduration=1.486811434 podStartE2EDuration="1.486811434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:10.486071534 +0000 UTC m=+1891.305809353" watchObservedRunningTime="2026-10-30 22:50:10.486811434 +0000 UTC m=+1891.306549252"
Oct 30 22:50:10.853589 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:10.850595    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:19.847251 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:19.847213    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:19.859455 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:19.859429    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:20.563714 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:20.563681    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:34.716427 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.716389    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"]
Oct 30 22:50:34.719897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.719876    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:50:34.730557 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.730524    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-8q7fd\""
Oct 30 22:50:34.730804 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.730532    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Oct 30 22:50:34.730928 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.730594    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Oct 30 22:50:34.730928 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.730625    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Oct 30 22:50:34.730928 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.730796    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-84m2z\""
Oct 30 22:50:34.731419 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.731400    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"]
Oct 30 22:50:34.776573 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:50:34.776536    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:50:34.846583 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.846547    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/476c18f6-5a46-43a1-bed0-758645b8fd7c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf\" (UID: \"476c18f6-5a46-43a1-bed0-758645b8fd7c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:50:34.846780 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.846718    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmftb\" (UniqueName: \"kubernetes.io/projected/476c18f6-5a46-43a1-bed0-758645b8fd7c-kube-api-access-pmftb\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf\" (UID: \"476c18f6-5a46-43a1-bed0-758645b8fd7c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:50:34.947907 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.947871    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmftb\" (UniqueName: \"kubernetes.io/projected/476c18f6-5a46-43a1-bed0-758645b8fd7c-kube-api-access-pmftb\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf\" (UID: \"476c18f6-5a46-43a1-bed0-758645b8fd7c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:50:34.947907 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.947911    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/476c18f6-5a46-43a1-bed0-758645b8fd7c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf\" (UID: \"476c18f6-5a46-43a1-bed0-758645b8fd7c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:50:34.950441 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.950408    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/476c18f6-5a46-43a1-bed0-758645b8fd7c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf\" (UID: \"476c18f6-5a46-43a1-bed0-758645b8fd7c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:50:34.971159 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:34.971087    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmftb\" (UniqueName: \"kubernetes.io/projected/476c18f6-5a46-43a1-bed0-758645b8fd7c-kube-api-access-pmftb\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf\" (UID: \"476c18f6-5a46-43a1-bed0-758645b8fd7c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:50:35.042978 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:35.042943    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:50:35.319060 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:35.318848    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"]
Oct 30 22:50:35.321776 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:50:35.321743    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476c18f6_5a46_43a1_bed0_758645b8fd7c.slice/crio-75bf1b913d893a12988c05b7c774c3adcceb101b1451a9192f64ab3e0eda2e7f WatchSource:0}: Error finding container 75bf1b913d893a12988c05b7c774c3adcceb101b1451a9192f64ab3e0eda2e7f: Status 404 returned error can't find the container with id 75bf1b913d893a12988c05b7c774c3adcceb101b1451a9192f64ab3e0eda2e7f
Oct 30 22:50:35.624768 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:35.624664    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf" event={"ID":"476c18f6-5a46-43a1-bed0-758645b8fd7c","Type":"ContainerStarted","Data":"2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220"}
Oct 30 22:50:35.625054 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:35.625008    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf" event={"ID":"476c18f6-5a46-43a1-bed0-758645b8fd7c","Type":"ContainerStarted","Data":"75bf1b913d893a12988c05b7c774c3adcceb101b1451a9192f64ab3e0eda2e7f"}
Oct 30 22:50:35.625442 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:35.625413    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:50:35.652907 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:35.652850    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf" podStartSLOduration=1.652833515 podStartE2EDuration="1.652833515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:35.650583514 +0000 UTC m=+1916.470321333" watchObservedRunningTime="2026-10-30 22:50:35.652833515 +0000 UTC m=+1916.472571333"
Oct 30 22:50:44.715864 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:44.715824    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"]
Oct 30 22:50:44.726404 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:50:44.726364    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61" cmd=["/bin/sleep","15"]
Oct 30 22:50:44.726628 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:44.726425    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:50:44.726628 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:50:44.726445    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg" podUID="f40a398d-2577-4ecc-9c4c-e1ff431a6144" containerName="main" containerID="cri-o://ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61"
Oct 30 22:50:44.726628 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:44.726516    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg" podUID="f40a398d-2577-4ecc-9c4c-e1ff431a6144" containerName="main" containerID="cri-o://ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61" gracePeriod=60
Oct 30 22:50:45.090494 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.090459    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:45.254310 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254222    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40a398d-2577-4ecc-9c4c-e1ff431a6144-kube-api-access-8jctk\" (UniqueName: \"kubernetes.io/projected/f40a398d-2577-4ecc-9c4c-e1ff431a6144-kube-api-access-8jctk\") pod \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") "
Oct 30 22:50:45.254310 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254296    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-home\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-home\") pod \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") "
Oct 30 22:50:45.254578 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254324    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-dshm\") pod \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") "
Oct 30 22:50:45.254578 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254342    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-model-cache\") pod \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") "
Oct 30 22:50:45.254578 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254375    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tmp-dir\") pod \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") "
Oct 30 22:50:45.254578 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254462    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tls-certs\" (UniqueName: \"kubernetes.io/secret/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tls-certs\") pod \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\" (UID: \"f40a398d-2577-4ecc-9c4c-e1ff431a6144\") "
Oct 30 22:50:45.254791 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254602    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-model-cache" pod "f40a398d-2577-4ecc-9c4c-e1ff431a6144" (UID: "f40a398d-2577-4ecc-9c4c-e1ff431a6144"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:45.254791 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254630    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-home" pod "f40a398d-2577-4ecc-9c4c-e1ff431a6144" (UID: "f40a398d-2577-4ecc-9c4c-e1ff431a6144"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:45.254906 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254816    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tmp-dir" pod "f40a398d-2577-4ecc-9c4c-e1ff431a6144" (UID: "f40a398d-2577-4ecc-9c4c-e1ff431a6144"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:45.254906 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254829    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:45.254906 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.254861    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:45.257090 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.257060    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tls-certs" pod "f40a398d-2577-4ecc-9c4c-e1ff431a6144" (UID: "f40a398d-2577-4ecc-9c4c-e1ff431a6144"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:45.257222 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.257178    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-dshm" pod "f40a398d-2577-4ecc-9c4c-e1ff431a6144" (UID: "f40a398d-2577-4ecc-9c4c-e1ff431a6144"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:45.257222 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.257189    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40a398d-2577-4ecc-9c4c-e1ff431a6144-kube-api-access-8jctk" pod "f40a398d-2577-4ecc-9c4c-e1ff431a6144" (UID: "f40a398d-2577-4ecc-9c4c-e1ff431a6144"). InnerVolumeSpecName "kube-api-access-8jctk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:45.355943 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.355908    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jctk\" (UniqueName: \"kubernetes.io/projected/f40a398d-2577-4ecc-9c4c-e1ff431a6144-kube-api-access-8jctk\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:45.355943 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.355938    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:45.355943 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.355948    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:45.356167 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.355956    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40a398d-2577-4ecc-9c4c-e1ff431a6144-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:45.671294 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.671195    5060 generic.go:356] "Generic (PLEG): container finished" podID="f40a398d-2577-4ecc-9c4c-e1ff431a6144" containerID="ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61" exitCode=0
Oct 30 22:50:45.671294 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.671228    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"
Oct 30 22:50:45.671294 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.671266    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg" event={"ID":"f40a398d-2577-4ecc-9c4c-e1ff431a6144","Type":"ContainerDied","Data":"ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61"}
Oct 30 22:50:45.671572 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.671321    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg" event={"ID":"f40a398d-2577-4ecc-9c4c-e1ff431a6144","Type":"ContainerDied","Data":"7b40e0b7d77e0a5dbc9c739e26a75e20aa839ffcffaaa4e069872b8c44e93927"}
Oct 30 22:50:45.671572 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.671345    5060 scope.go:122] "RemoveContainer" containerID="ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61"
Oct 30 22:50:45.680321 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.680298    5060 scope.go:122] "RemoveContainer" containerID="ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61"
Oct 30 22:50:45.680589 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:50:45.680570    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61\": container with ID starting with ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61 not found: ID does not exist" containerID="ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61"
Oct 30 22:50:45.680649 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.680596    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61"} err="failed to get container status \"ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61\": rpc error: code = NotFound desc = could not find container \"ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61\": container with ID starting with ddd499a147ff86c2654d7b7b019474b89228ce2d2ff0aa7baa1e410b3f253e61 not found: ID does not exist"
Oct 30 22:50:45.694930 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.692041    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"]
Oct 30 22:50:45.697744 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:45.697720    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-vnftg"]
Oct 30 22:50:47.289624 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:50:47.289590    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40a398d-2577-4ecc-9c4c-e1ff431a6144" path="/var/lib/kubelet/pods/f40a398d-2577-4ecc-9c4c-e1ff431a6144/volumes"
Oct 30 22:51:04.775498 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:51:04.775415    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:51:06.632095 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:51:06.632062    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:51:34.775469 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:51:34.775429    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:52:04.776262 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:52:04.776227    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:52:16.545979 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.545942    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"]
Oct 30 22:52:16.546457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.546345    5060 cpu_manager.go:414] "containerMap: removing container" podUID="f40a398d-2577-4ecc-9c4c-e1ff431a6144" containerName="main"
Oct 30 22:52:16.546457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.546358    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40a398d-2577-4ecc-9c4c-e1ff431a6144" containerName="main"
Oct 30 22:52:16.546457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.546443    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40a398d-2577-4ecc-9c4c-e1ff431a6144" containerName="main"
Oct 30 22:52:16.549563 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.549543    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.559600 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.559568    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Oct 30 22:52:16.559600 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.559592    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Oct 30 22:52:16.559762 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.559616    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-jgwjb\""
Oct 30 22:52:16.562292 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.562277    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Oct 30 22:52:16.570225 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.570200    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"]
Oct 30 22:52:16.665468 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.665425    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.665468 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.665470    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.665759 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.665493    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.665759 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.665620    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mxgs\" (UniqueName: \"kubernetes.io/projected/3689bc95-5758-4ff0-8c64-5d703f1fcd10-kube-api-access-4mxgs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.665759 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.665674    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.665759 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.665711    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.766241 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.766188    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.766241 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.766252    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxgs\" (UniqueName: \"kubernetes.io/projected/3689bc95-5758-4ff0-8c64-5d703f1fcd10-kube-api-access-4mxgs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.766484 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.766304    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.766484 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.766345    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.766484 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.766422    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.766484 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.766462    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.766759 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.766734    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.766873 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.766768    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.766873 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.766805    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.768774 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.768748    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.768979 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.768963    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.780569 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.780540    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxgs\" (UniqueName: \"kubernetes.io/projected/3689bc95-5758-4ff0-8c64-5d703f1fcd10-kube-api-access-4mxgs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:16.862278 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:16.862185    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:17.163279 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:17.163251    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"]
Oct 30 22:52:17.164800 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:52:17.164761    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3689bc95_5758_4ff0_8c64_5d703f1fcd10.slice/crio-e08f732d2e53a02cb86056c0dd31f02dbac25621e74e99f50f036d12b6c54c44 WatchSource:0}: Error finding container e08f732d2e53a02cb86056c0dd31f02dbac25621e74e99f50f036d12b6c54c44: Status 404 returned error can't find the container with id e08f732d2e53a02cb86056c0dd31f02dbac25621e74e99f50f036d12b6c54c44
Oct 30 22:52:18.095715 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:18.095676    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h" event={"ID":"3689bc95-5758-4ff0-8c64-5d703f1fcd10","Type":"ContainerStarted","Data":"f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979"}
Oct 30 22:52:18.096221 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:18.095740    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h" event={"ID":"3689bc95-5758-4ff0-8c64-5d703f1fcd10","Type":"ContainerStarted","Data":"e08f732d2e53a02cb86056c0dd31f02dbac25621e74e99f50f036d12b6c54c44"}
Oct 30 22:52:18.125835 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:18.125776    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h" podStartSLOduration=2.125761438 podStartE2EDuration="2.125761438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:18.123568035 +0000 UTC m=+2018.943305853" watchObservedRunningTime="2026-10-30 22:52:18.125761438 +0000 UTC m=+2018.945499256"
Oct 30 22:52:18.862847 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:18.862808    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:26.863343 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:26.863250    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:26.875173 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:26.875147    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:27.147541 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:27.147444    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:34.775208 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:52:34.775165    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:52:41.024186 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:41.024138    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" podUID="a5c727ac326546386befce51c4d99c6d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:52:50.093601 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.093559    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"]
Oct 30 22:52:50.104228 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:52:50.104186    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979" cmd=["/bin/sleep","15"]
Oct 30 22:52:50.104400 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.104246    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 22:52:50.104400 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:52:50.104260    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h" podUID="3689bc95-5758-4ff0-8c64-5d703f1fcd10" containerName="main" containerID="cri-o://f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979"
Oct 30 22:52:50.104400 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.104315    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h" podUID="3689bc95-5758-4ff0-8c64-5d703f1fcd10" containerName="main" containerID="cri-o://f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979" gracePeriod=60
Oct 30 22:52:50.137712 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.137673    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h" podUID="3689bc95-5758-4ff0-8c64-5d703f1fcd10" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Oct 30 22:52:50.454235 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.454209    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:50.534237 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.534199    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3689bc95-5758-4ff0-8c64-5d703f1fcd10-kube-api-access-4mxgs\" (UniqueName: \"kubernetes.io/projected/3689bc95-5758-4ff0-8c64-5d703f1fcd10-kube-api-access-4mxgs\") pod \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") "
Oct 30 22:52:50.534401 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.534268    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-home\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-home\") pod \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") "
Oct 30 22:52:50.534401 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.534344    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tls-certs\" (UniqueName: \"kubernetes.io/secret/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tls-certs\") pod \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") "
Oct 30 22:52:50.534401 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.534378    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-dshm\") pod \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") "
Oct 30 22:52:50.534564 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.534419    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tmp-dir\") pod \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") "
Oct 30 22:52:50.534564 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.534454    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-model-cache\") pod \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\" (UID: \"3689bc95-5758-4ff0-8c64-5d703f1fcd10\") "
Oct 30 22:52:50.534676 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.534602    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-home" pod "3689bc95-5758-4ff0-8c64-5d703f1fcd10" (UID: "3689bc95-5758-4ff0-8c64-5d703f1fcd10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:50.534794 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.534768    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tmp-dir" pod "3689bc95-5758-4ff0-8c64-5d703f1fcd10" (UID: "3689bc95-5758-4ff0-8c64-5d703f1fcd10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:50.534855 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.534785    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-model-cache" pod "3689bc95-5758-4ff0-8c64-5d703f1fcd10" (UID: "3689bc95-5758-4ff0-8c64-5d703f1fcd10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:50.534855 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.534828    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:50.536637 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.536617    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tls-certs" pod "3689bc95-5758-4ff0-8c64-5d703f1fcd10" (UID: "3689bc95-5758-4ff0-8c64-5d703f1fcd10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:50.536844 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.536826    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-dshm" pod "3689bc95-5758-4ff0-8c64-5d703f1fcd10" (UID: "3689bc95-5758-4ff0-8c64-5d703f1fcd10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:50.536930 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.536910    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3689bc95-5758-4ff0-8c64-5d703f1fcd10-kube-api-access-4mxgs" pod "3689bc95-5758-4ff0-8c64-5d703f1fcd10" (UID: "3689bc95-5758-4ff0-8c64-5d703f1fcd10"). InnerVolumeSpecName "kube-api-access-4mxgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:50.635897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.635801    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:50.635897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.635836    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:50.635897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.635849    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:50.635897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.635860    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3689bc95-5758-4ff0-8c64-5d703f1fcd10-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:50.635897 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:50.635877    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mxgs\" (UniqueName: \"kubernetes.io/projected/3689bc95-5758-4ff0-8c64-5d703f1fcd10-kube-api-access-4mxgs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:51.254024 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:51.253983    5060 generic.go:356] "Generic (PLEG): container finished" podID="3689bc95-5758-4ff0-8c64-5d703f1fcd10" containerID="f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979" exitCode=0
Oct 30 22:52:51.254423 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:51.254033    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"
Oct 30 22:52:51.254423 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:51.254040    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h" event={"ID":"3689bc95-5758-4ff0-8c64-5d703f1fcd10","Type":"ContainerDied","Data":"f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979"}
Oct 30 22:52:51.254423 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:51.254095    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h" event={"ID":"3689bc95-5758-4ff0-8c64-5d703f1fcd10","Type":"ContainerDied","Data":"e08f732d2e53a02cb86056c0dd31f02dbac25621e74e99f50f036d12b6c54c44"}
Oct 30 22:52:51.254423 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:51.254126    5060 scope.go:122] "RemoveContainer" containerID="f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979"
Oct 30 22:52:51.263721 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:51.263703    5060 scope.go:122] "RemoveContainer" containerID="f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979"
Oct 30 22:52:51.263987 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:52:51.263960    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979\": container with ID starting with f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979 not found: ID does not exist" containerID="f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979"
Oct 30 22:52:51.264070 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:51.264001    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979"} err="failed to get container status \"f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979\": rpc error: code = NotFound desc = could not find container \"f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979\": container with ID starting with f6467386cae5d648ae8bf8b24466096680a72b048318bd479a240a43eadba979 not found: ID does not exist"
Oct 30 22:52:51.291149 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:51.291123    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"]
Oct 30 22:52:51.305111 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:51.305084    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584bpf62h"]
Oct 30 22:52:53.289331 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:53.289298    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3689bc95-5758-4ff0-8c64-5d703f1fcd10" path="/var/lib/kubelet/pods/3689bc95-5758-4ff0-8c64-5d703f1fcd10/volumes"
Oct 30 22:52:56.902284 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:52:56.902250    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"]
Oct 30 22:53:04.775791 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:53:04.775755    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:53:06.631952 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:06.631916    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf" podUID="476c18f6-5a46-43a1-bed0-758645b8fd7c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:53:11.922638 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:11.922578    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf" podUID="476c18f6-5a46-43a1-bed0-758645b8fd7c" containerName="main" containerID="cri-o://2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220" gracePeriod=45
Oct 30 22:53:16.632154 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:16.632112    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf" podUID="476c18f6-5a46-43a1-bed0-758645b8fd7c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:53:26.631718 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:26.631677    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf" podUID="476c18f6-5a46-43a1-bed0-758645b8fd7c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:53:26.632268 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:26.631793    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:53:34.775137 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:53:34.775097    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:53:36.631769 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:36.631732    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf" podUID="476c18f6-5a46-43a1-bed0-758645b8fd7c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:53:39.561945 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:39.561916    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:53:39.565227 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:39.565202    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:53:42.182735 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.182701    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:53:42.293111 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.293071    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/476c18f6-5a46-43a1-bed0-758645b8fd7c-tls-certs\" (UniqueName: \"kubernetes.io/secret/476c18f6-5a46-43a1-bed0-758645b8fd7c-tls-certs\") pod \"476c18f6-5a46-43a1-bed0-758645b8fd7c\" (UID: \"476c18f6-5a46-43a1-bed0-758645b8fd7c\") "
Oct 30 22:53:42.293111 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.293111    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476c18f6-5a46-43a1-bed0-758645b8fd7c-kube-api-access-pmftb\" (UniqueName: \"kubernetes.io/projected/476c18f6-5a46-43a1-bed0-758645b8fd7c-kube-api-access-pmftb\") pod \"476c18f6-5a46-43a1-bed0-758645b8fd7c\" (UID: \"476c18f6-5a46-43a1-bed0-758645b8fd7c\") "
Oct 30 22:53:42.295357 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.295323    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476c18f6-5a46-43a1-bed0-758645b8fd7c-tls-certs" pod "476c18f6-5a46-43a1-bed0-758645b8fd7c" (UID: "476c18f6-5a46-43a1-bed0-758645b8fd7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:42.295481 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.295380    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476c18f6-5a46-43a1-bed0-758645b8fd7c-kube-api-access-pmftb" pod "476c18f6-5a46-43a1-bed0-758645b8fd7c" (UID: "476c18f6-5a46-43a1-bed0-758645b8fd7c"). InnerVolumeSpecName "kube-api-access-pmftb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:42.394133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.394090    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/476c18f6-5a46-43a1-bed0-758645b8fd7c-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:42.394133 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.394129    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmftb\" (UniqueName: \"kubernetes.io/projected/476c18f6-5a46-43a1-bed0-758645b8fd7c-kube-api-access-pmftb\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:42.493477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.493441    5060 generic.go:356] "Generic (PLEG): container finished" podID="476c18f6-5a46-43a1-bed0-758645b8fd7c" containerID="2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220" exitCode=0
Oct 30 22:53:42.493679 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.493485    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"
Oct 30 22:53:42.493679 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.493529    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf" event={"ID":"476c18f6-5a46-43a1-bed0-758645b8fd7c","Type":"ContainerDied","Data":"2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220"}
Oct 30 22:53:42.493679 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.493584    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf" event={"ID":"476c18f6-5a46-43a1-bed0-758645b8fd7c","Type":"ContainerDied","Data":"75bf1b913d893a12988c05b7c774c3adcceb101b1451a9192f64ab3e0eda2e7f"}
Oct 30 22:53:42.493679 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.493606    5060 scope.go:122] "RemoveContainer" containerID="2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220"
Oct 30 22:53:42.502656 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.502624    5060 scope.go:122] "RemoveContainer" containerID="2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220"
Oct 30 22:53:42.502894 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:53:42.502876    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220\": container with ID starting with 2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220 not found: ID does not exist" containerID="2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220"
Oct 30 22:53:42.502938 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.502902    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220"} err="failed to get container status \"2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220\": rpc error: code = NotFound desc = could not find container \"2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220\": container with ID starting with 2765bf173ecf314bd2a29aec85eadb60ab4e38b98dc8bb628ec95ac626d51220 not found: ID does not exist"
Oct 30 22:53:42.516664 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.516641    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"]
Oct 30 22:53:42.521342 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:42.521318    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c6894sgpjf"]
Oct 30 22:53:43.289130 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:43.289095    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476c18f6-5a46-43a1-bed0-758645b8fd7c" path="/var/lib/kubelet/pods/476c18f6-5a46-43a1-bed0-758645b8fd7c/volumes"
Oct 30 22:53:50.567741 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.567649    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"]
Oct 30 22:53:50.568120 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.568012    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3689bc95-5758-4ff0-8c64-5d703f1fcd10" containerName="main"
Oct 30 22:53:50.568120 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.568022    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3689bc95-5758-4ff0-8c64-5d703f1fcd10" containerName="main"
Oct 30 22:53:50.568120 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.568037    5060 cpu_manager.go:414] "containerMap: removing container" podUID="476c18f6-5a46-43a1-bed0-758645b8fd7c" containerName="main"
Oct 30 22:53:50.568120 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.568042    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476c18f6-5a46-43a1-bed0-758645b8fd7c" containerName="main"
Oct 30 22:53:50.568120 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.568095    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="476c18f6-5a46-43a1-bed0-758645b8fd7c" containerName="main"
Oct 30 22:53:50.568120 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.568109    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3689bc95-5758-4ff0-8c64-5d703f1fcd10" containerName="main"
Oct 30 22:53:50.572582 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.572561    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.577647 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.577623    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Oct 30 22:53:50.578288 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.578258    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Oct 30 22:53:50.578398 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.578316    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:53:50.578398 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.578357    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-rvvkp\""
Oct 30 22:53:50.578398 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.578375    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Oct 30 22:53:50.578398 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.578391    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Oct 30 22:53:50.584831 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.584807    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"]
Oct 30 22:53:50.667184 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.667131    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-dshm\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.667184 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.667193    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-home\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.667408 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.667292    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tmp-dir\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.667408 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.667331    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqlfm\" (UniqueName: \"kubernetes.io/projected/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kube-api-access-rqlfm\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.667408 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.667403    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kserve-provision-location\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.667535 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.667428    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-model-cache\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.667535 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.667468    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tls-certs\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.667620 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.667539    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ade8083-7828-478f-9e6c-ca9c6f9a0800-cabundle-cert\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.768685 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.768626    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kserve-provision-location\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.768685 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.768687    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-model-cache\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.768951 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.768709    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tls-certs\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.768951 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.768746    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ade8083-7828-478f-9e6c-ca9c6f9a0800-cabundle-cert\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.768951 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.768775    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-dshm\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.768951 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.768806    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-home\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.768951 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.768829    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tmp-dir\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.768951 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.768854    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqlfm\" (UniqueName: \"kubernetes.io/projected/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kube-api-access-rqlfm\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.769230 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.769100    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kserve-provision-location\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.769230 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.769155    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-model-cache\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.769301 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.769258    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tmp-dir\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.769376 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.769354    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-home\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.769568 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.769551    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ade8083-7828-478f-9e6c-ca9c6f9a0800-cabundle-cert\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.771139 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.771118    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-dshm\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.771402 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.771388    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tls-certs\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.781518 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.781483    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqlfm\" (UniqueName: \"kubernetes.io/projected/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kube-api-access-rqlfm\") pod \"auth-disabled-test-kserve-597d55d8b8-v5hr8\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.820196 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.820112    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"]
Oct 30 22:53:50.825669 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.825646    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:53:50.832662 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.832637    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-q2llp\""
Oct 30 22:53:50.833927 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.833903    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"]
Oct 30 22:53:50.870273 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.870233    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46c4606-573a-4f10-b142-9d3f54385fc5-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h\" (UID: \"a46c4606-573a-4f10-b142-9d3f54385fc5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:53:50.870454 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.870380    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlk9t\" (UniqueName: \"kubernetes.io/projected/a46c4606-573a-4f10-b142-9d3f54385fc5-kube-api-access-vlk9t\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h\" (UID: \"a46c4606-573a-4f10-b142-9d3f54385fc5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:53:50.888206 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.888173    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:53:50.971437 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.971399    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlk9t\" (UniqueName: \"kubernetes.io/projected/a46c4606-573a-4f10-b142-9d3f54385fc5-kube-api-access-vlk9t\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h\" (UID: \"a46c4606-573a-4f10-b142-9d3f54385fc5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:53:50.971593 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.971482    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46c4606-573a-4f10-b142-9d3f54385fc5-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h\" (UID: \"a46c4606-573a-4f10-b142-9d3f54385fc5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:53:50.973943 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.973921    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46c4606-573a-4f10-b142-9d3f54385fc5-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h\" (UID: \"a46c4606-573a-4f10-b142-9d3f54385fc5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:53:50.984250 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:50.984227    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlk9t\" (UniqueName: \"kubernetes.io/projected/a46c4606-573a-4f10-b142-9d3f54385fc5-kube-api-access-vlk9t\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h\" (UID: \"a46c4606-573a-4f10-b142-9d3f54385fc5\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:53:51.141637 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:51.141562    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:53:51.174379 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:51.174336    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"]
Oct 30 22:53:51.177485 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:53:51.177456    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ade8083_7828_478f_9e6c_ca9c6f9a0800.slice/crio-ce493d213c91d6252c847217db3af9c2281d66fc7fe076db1533907c4552165d WatchSource:0}: Error finding container ce493d213c91d6252c847217db3af9c2281d66fc7fe076db1533907c4552165d: Status 404 returned error can't find the container with id ce493d213c91d6252c847217db3af9c2281d66fc7fe076db1533907c4552165d
Oct 30 22:53:51.422490 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:51.422458    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"]
Oct 30 22:53:51.425449 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:53:51.425417    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46c4606_573a_4f10_b142_9d3f54385fc5.slice/crio-ac5d6ab794cc9602554242c76e51c503e723c120b69edc975aca8d9688d442fe WatchSource:0}: Error finding container ac5d6ab794cc9602554242c76e51c503e723c120b69edc975aca8d9688d442fe: Status 404 returned error can't find the container with id ac5d6ab794cc9602554242c76e51c503e723c120b69edc975aca8d9688d442fe
Oct 30 22:53:51.539789 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:51.539744    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h" event={"ID":"a46c4606-573a-4f10-b142-9d3f54385fc5","Type":"ContainerStarted","Data":"ac5d6ab794cc9602554242c76e51c503e723c120b69edc975aca8d9688d442fe"}
Oct 30 22:53:51.541481 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:51.541450    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" event={"ID":"2ade8083-7828-478f-9e6c-ca9c6f9a0800","Type":"ContainerStarted","Data":"8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555"}
Oct 30 22:53:51.541624 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:51.541497    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" event={"ID":"2ade8083-7828-478f-9e6c-ca9c6f9a0800","Type":"ContainerStarted","Data":"ce493d213c91d6252c847217db3af9c2281d66fc7fe076db1533907c4552165d"}
Oct 30 22:53:52.548061 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:52.548017    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h" event={"ID":"a46c4606-573a-4f10-b142-9d3f54385fc5","Type":"ContainerStarted","Data":"cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7"}
Oct 30 22:53:52.548603 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:52.548311    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:53:52.570458 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:52.570398    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h" podStartSLOduration=2.570383036 podStartE2EDuration="2.570383036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:52.569137973 +0000 UTC m=+2113.388875786" watchObservedRunningTime="2026-10-30 22:53:52.570383036 +0000 UTC m=+2113.390120855"
Oct 30 22:53:54.558682 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:54.558645    5060 generic.go:356] "Generic (PLEG): container finished" podID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerID="8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555" exitCode=0
Oct 30 22:53:54.559065 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:54.558700    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" event={"ID":"2ade8083-7828-478f-9e6c-ca9c6f9a0800","Type":"ContainerDied","Data":"8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555"}
Oct 30 22:53:55.566322 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:55.566285    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" event={"ID":"2ade8083-7828-478f-9e6c-ca9c6f9a0800","Type":"ContainerStarted","Data":"8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b"}
Oct 30 22:53:55.599785 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:55.599726    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podStartSLOduration=3.133910025 podStartE2EDuration="5.599708352s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.465798327s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:55.597811401 +0000 UTC m=+2116.417549232" watchObservedRunningTime="2026-10-30 22:53:55.599708352 +0000 UTC m=+2116.419446173"
Oct 30 22:53:55.888831 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:53:55.888746    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:54:00.888709 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:54:00.888657    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:54:00.890350 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:54:00.890318    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:54:04.777036 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:54:04.776990    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:54:10.889283 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:54:10.889228    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:54:20.889160 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:54:20.889122    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:54:23.572210 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:54:23.571722    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:54:30.888608 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:54:30.888562    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:54:34.778003 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:54:34.777969    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:54:40.889128 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:54:40.889086    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:54:50.889325 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:54:50.889283    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:55:00.888663 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:55:00.888610    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:55:04.775601 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:55:04.775560    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:55:10.889102 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:55:10.889061    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:55:20.888759 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:55:20.888664    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:55:30.889068 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:55:30.889018    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:55:34.776207 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:55:34.776160    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:55:40.888997 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:55:40.888950    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:55:50.898585 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:55:50.898549    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:55:50.906441 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:55:50.906414    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:55:55.282264 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:55:55.282228    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"]
Oct 30 22:55:55.291846 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:55:55.291819    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"]
Oct 30 22:56:03.554461 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:03.554421    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h" podUID="a46c4606-573a-4f10-b142-9d3f54385fc5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:04.774773 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:56:04.774734    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:56:10.308668 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:10.308608    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" containerID="cri-o://8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b" gracePeriod=45
Oct 30 22:56:10.311523 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:10.311478    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h" podUID="a46c4606-573a-4f10-b142-9d3f54385fc5" containerName="main" containerID="cri-o://cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7" gracePeriod=45
Oct 30 22:56:10.899989 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:10.899941    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Oct 30 22:56:11.059274 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.059250    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:56:11.156995 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.156902    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tmp-dir\") pod \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") "
Oct 30 22:56:11.156995 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.156944    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-model-cache\") pod \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") "
Oct 30 22:56:11.156995 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.156972    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ade8083-7828-478f-9e6c-ca9c6f9a0800-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ade8083-7828-478f-9e6c-ca9c6f9a0800-cabundle-cert\") pod \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") "
Oct 30 22:56:11.157277 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.157008    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-home\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-home\") pod \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") "
Oct 30 22:56:11.157277 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.157032    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-dshm\") pod \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") "
Oct 30 22:56:11.157277 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.157080    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kube-api-access-rqlfm\" (UniqueName: \"kubernetes.io/projected/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kube-api-access-rqlfm\") pod \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") "
Oct 30 22:56:11.157277 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.157112    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tls-certs\") pod \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") "
Oct 30 22:56:11.157277 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.157147    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kserve-provision-location\") pod \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\" (UID: \"2ade8083-7828-478f-9e6c-ca9c6f9a0800\") "
Oct 30 22:56:11.157277 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.157215    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-model-cache" pod "2ade8083-7828-478f-9e6c-ca9c6f9a0800" (UID: "2ade8083-7828-478f-9e6c-ca9c6f9a0800"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:11.157613 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.157360    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ade8083-7828-478f-9e6c-ca9c6f9a0800-cabundle-cert" pod "2ade8083-7828-478f-9e6c-ca9c6f9a0800" (UID: "2ade8083-7828-478f-9e6c-ca9c6f9a0800"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:11.157613 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.157489    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:11.157729 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.157655    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ade8083-7828-478f-9e6c-ca9c6f9a0800-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:11.157869 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.157839    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-home" pod "2ade8083-7828-478f-9e6c-ca9c6f9a0800" (UID: "2ade8083-7828-478f-9e6c-ca9c6f9a0800"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:11.159424 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.159395    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tls-certs" pod "2ade8083-7828-478f-9e6c-ca9c6f9a0800" (UID: "2ade8083-7828-478f-9e6c-ca9c6f9a0800"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:11.159555 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.159465    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-dshm" pod "2ade8083-7828-478f-9e6c-ca9c6f9a0800" (UID: "2ade8083-7828-478f-9e6c-ca9c6f9a0800"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:11.159555 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.159490    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kube-api-access-rqlfm" pod "2ade8083-7828-478f-9e6c-ca9c6f9a0800" (UID: "2ade8083-7828-478f-9e6c-ca9c6f9a0800"). InnerVolumeSpecName "kube-api-access-rqlfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:11.169117 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.169093    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tmp-dir" pod "2ade8083-7828-478f-9e6c-ca9c6f9a0800" (UID: "2ade8083-7828-478f-9e6c-ca9c6f9a0800"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:11.186942 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.186908    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kserve-provision-location" pod "2ade8083-7828-478f-9e6c-ca9c6f9a0800" (UID: "2ade8083-7828-478f-9e6c-ca9c6f9a0800"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:11.244940 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.244901    5060 generic.go:356] "Generic (PLEG): container finished" podID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerID="8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b" exitCode=0
Oct 30 22:56:11.245125 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.244953    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"
Oct 30 22:56:11.245125 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.244958    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" event={"ID":"2ade8083-7828-478f-9e6c-ca9c6f9a0800","Type":"ContainerDied","Data":"8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b"}
Oct 30 22:56:11.245125 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.245071    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8" event={"ID":"2ade8083-7828-478f-9e6c-ca9c6f9a0800","Type":"ContainerDied","Data":"ce493d213c91d6252c847217db3af9c2281d66fc7fe076db1533907c4552165d"}
Oct 30 22:56:11.245125 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.245091    5060 scope.go:122] "RemoveContainer" containerID="8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b"
Oct 30 22:56:11.255099 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.255076    5060 scope.go:122] "RemoveContainer" containerID="8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555"
Oct 30 22:56:11.258363 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.258344    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:11.258492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.258368    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:11.258492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.258381    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:11.258492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.258395    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqlfm\" (UniqueName: \"kubernetes.io/projected/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kube-api-access-rqlfm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:11.258492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.258409    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ade8083-7828-478f-9e6c-ca9c6f9a0800-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:11.258492 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.258424    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ade8083-7828-478f-9e6c-ca9c6f9a0800-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:11.263046 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.263023    5060 scope.go:122] "RemoveContainer" containerID="8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b"
Oct 30 22:56:11.263312 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:56:11.263294    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b\": container with ID starting with 8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b not found: ID does not exist" containerID="8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b"
Oct 30 22:56:11.263384 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.263324    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b"} err="failed to get container status \"8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b\": rpc error: code = NotFound desc = could not find container \"8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b\": container with ID starting with 8c8bc0e3a1be1f47c233ae554a5c6bfa3c6562f55ac3ae208a64e03c13f6498b not found: ID does not exist"
Oct 30 22:56:11.263384 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.263350    5060 scope.go:122] "RemoveContainer" containerID="8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555"
Oct 30 22:56:11.263642 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:56:11.263624    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555\": container with ID starting with 8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555 not found: ID does not exist" containerID="8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555"
Oct 30 22:56:11.263679 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.263649    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555"} err="failed to get container status \"8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555\": rpc error: code = NotFound desc = could not find container \"8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555\": container with ID starting with 8fb7c1e3328911c0425fe65993fe7bac399ecb657d3cbab2e256334fd81b8555 not found: ID does not exist"
Oct 30 22:56:11.270573 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.270551    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"]
Oct 30 22:56:11.275642 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.275620    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-597d55d8b8-v5hr8"]
Oct 30 22:56:11.289751 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:11.289728    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" path="/var/lib/kubelet/pods/2ade8083-7828-478f-9e6c-ca9c6f9a0800/volumes"
Oct 30 22:56:13.554523 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:13.554467    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h" podUID="a46c4606-573a-4f10-b142-9d3f54385fc5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:23.554610 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:23.554566    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h" podUID="a46c4606-573a-4f10-b142-9d3f54385fc5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:23.555161 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:23.554693    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:56:33.554242 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:33.554204    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h" podUID="a46c4606-573a-4f10-b142-9d3f54385fc5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:34.775223 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:56:34.775184    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:56:40.565093 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:40.565067    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:56:40.725613 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:40.725580    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a46c4606-573a-4f10-b142-9d3f54385fc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/a46c4606-573a-4f10-b142-9d3f54385fc5-tls-certs\") pod \"a46c4606-573a-4f10-b142-9d3f54385fc5\" (UID: \"a46c4606-573a-4f10-b142-9d3f54385fc5\") "
Oct 30 22:56:40.725779 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:40.725630    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a46c4606-573a-4f10-b142-9d3f54385fc5-kube-api-access-vlk9t\" (UniqueName: \"kubernetes.io/projected/a46c4606-573a-4f10-b142-9d3f54385fc5-kube-api-access-vlk9t\") pod \"a46c4606-573a-4f10-b142-9d3f54385fc5\" (UID: \"a46c4606-573a-4f10-b142-9d3f54385fc5\") "
Oct 30 22:56:40.727639 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:40.727608    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46c4606-573a-4f10-b142-9d3f54385fc5-tls-certs" pod "a46c4606-573a-4f10-b142-9d3f54385fc5" (UID: "a46c4606-573a-4f10-b142-9d3f54385fc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:40.727753 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:40.727656    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46c4606-573a-4f10-b142-9d3f54385fc5-kube-api-access-vlk9t" pod "a46c4606-573a-4f10-b142-9d3f54385fc5" (UID: "a46c4606-573a-4f10-b142-9d3f54385fc5"). InnerVolumeSpecName "kube-api-access-vlk9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:40.826797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:40.826757    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46c4606-573a-4f10-b142-9d3f54385fc5-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:40.826797 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:40.826790    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlk9t\" (UniqueName: \"kubernetes.io/projected/a46c4606-573a-4f10-b142-9d3f54385fc5-kube-api-access-vlk9t\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:41.379997 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:41.379961    5060 generic.go:356] "Generic (PLEG): container finished" podID="a46c4606-573a-4f10-b142-9d3f54385fc5" containerID="cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7" exitCode=0
Oct 30 22:56:41.380178 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:41.380014    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"
Oct 30 22:56:41.380178 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:41.380047    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h" event={"ID":"a46c4606-573a-4f10-b142-9d3f54385fc5","Type":"ContainerDied","Data":"cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7"}
Oct 30 22:56:41.380178 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:41.380103    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h" event={"ID":"a46c4606-573a-4f10-b142-9d3f54385fc5","Type":"ContainerDied","Data":"ac5d6ab794cc9602554242c76e51c503e723c120b69edc975aca8d9688d442fe"}
Oct 30 22:56:41.380178 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:41.380124    5060 scope.go:122] "RemoveContainer" containerID="cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7"
Oct 30 22:56:41.388631 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:41.388612    5060 scope.go:122] "RemoveContainer" containerID="cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7"
Oct 30 22:56:41.388980 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:56:41.388953    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7\": container with ID starting with cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7 not found: ID does not exist" containerID="cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7"
Oct 30 22:56:41.389064 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:41.388989    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7"} err="failed to get container status \"cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7\": rpc error: code = NotFound desc = could not find container \"cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7\": container with ID starting with cf0a6a7eea9a0a8036591a9f2a93e6cbaf6b86913d9f7ba2afa91fa9c5fe50d7 not found: ID does not exist"
Oct 30 22:56:41.399352 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:41.399327    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"]
Oct 30 22:56:41.401919 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:41.401894    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-c622h"]
Oct 30 22:56:43.289174 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:43.289134    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a46c4606-573a-4f10-b142-9d3f54385fc5" path="/var/lib/kubelet/pods/a46c4606-573a-4f10-b142-9d3f54385fc5/volumes"
Oct 30 22:56:46.052894 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.052862    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"]
Oct 30 22:56:46.053381 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.053361    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="storage-initializer"
Oct 30 22:56:46.053457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.053384    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="storage-initializer"
Oct 30 22:56:46.053457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.053421    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a46c4606-573a-4f10-b142-9d3f54385fc5" containerName="main"
Oct 30 22:56:46.053457 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.053429    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46c4606-573a-4f10-b142-9d3f54385fc5" containerName="main"
Oct 30 22:56:46.053696 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.053460    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main"
Oct 30 22:56:46.053696 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.053469    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main"
Oct 30 22:56:46.053696 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.053584    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a46c4606-573a-4f10-b142-9d3f54385fc5" containerName="main"
Oct 30 22:56:46.053696 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.053599    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ade8083-7828-478f-9e6c-ca9c6f9a0800" containerName="main"
Oct 30 22:56:46.058010 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.057989    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.063416 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.063396    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:56:46.063553 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.063499    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Oct 30 22:56:46.064577 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.064552    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Oct 30 22:56:46.064689 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.064637    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-btfbq\""
Oct 30 22:56:46.082916 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.082886    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"]
Oct 30 22:56:46.173632 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.173597    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-podinfo\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.173815 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.173638    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-credential-socket\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.173815 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.173674    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-socket\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.173902 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.173814    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-envoy\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.173902 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.173852    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db4962f-1f52-42d2-94be-91317b48cb8a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.173966 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.173911    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-token\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.173966 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.173938    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-certs\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.174024 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.173986    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-data\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.174024 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.174018    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhsz5\" (UniqueName: \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-kube-api-access-jhsz5\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.274846 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.274809    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-envoy\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275042 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.274854    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db4962f-1f52-42d2-94be-91317b48cb8a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275042 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.274878    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-token\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275042 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.274898    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-certs\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275042 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.274914    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-data\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275042 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.275026    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhsz5\" (UniqueName: \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-kube-api-access-jhsz5\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.275094    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-podinfo\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.275132    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-credential-socket\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.275195    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-socket\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275316 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.275307    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-data\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275567 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.275390    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-certs\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275567 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.275496    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-credential-socket\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275567 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.275499    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-socket\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.275744 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.275652    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db4962f-1f52-42d2-94be-91317b48cb8a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.277227 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.277196    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-envoy\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.277451 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.277435    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-podinfo\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.290483 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.290453    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-token\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.290725 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.290690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhsz5\" (UniqueName: \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-kube-api-access-jhsz5\") pod \"router-gateway-1-openshift-default-64d7d8f984-998gb\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.371572 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.371533    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:46.654083 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.654059    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"]
Oct 30 22:56:46.656813 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:56:46.656786    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db4962f_1f52_42d2_94be_91317b48cb8a.slice/crio-057ced130b1e0c2b9f524de977c1e109ab03dbd38b8b63876c8a4fec92c09d2e WatchSource:0}: Error finding container 057ced130b1e0c2b9f524de977c1e109ab03dbd38b8b63876c8a4fec92c09d2e: Status 404 returned error can't find the container with id 057ced130b1e0c2b9f524de977c1e109ab03dbd38b8b63876c8a4fec92c09d2e
Oct 30 22:56:46.658776 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.658743    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:56:46.658826 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.658810    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:56:46.658858 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:46.658841    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:56:47.409290 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:47.409255    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" event={"ID":"4db4962f-1f52-42d2-94be-91317b48cb8a","Type":"ContainerStarted","Data":"b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5"}
Oct 30 22:56:47.409729 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:47.409310    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" event={"ID":"4db4962f-1f52-42d2-94be-91317b48cb8a","Type":"ContainerStarted","Data":"057ced130b1e0c2b9f524de977c1e109ab03dbd38b8b63876c8a4fec92c09d2e"}
Oct 30 22:56:48.371896 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:48.371858    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:48.376582 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:48.376556    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:48.401789 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:48.401733    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" podStartSLOduration=2.401719542 podStartE2EDuration="2.401719542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:47.436328809 +0000 UTC m=+2288.256066627" watchObservedRunningTime="2026-10-30 22:56:48.401719542 +0000 UTC m=+2289.221457361"
Oct 30 22:56:48.413227 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:48.413200    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:56:48.414089 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:56:48.414071    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:57:04.455679 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:04.455627    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:57:04.511108 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:04.511078    5060 generic.go:356] "Generic (PLEG): container finished" podID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerID="b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5" exitCode=0
Oct 30 22:57:04.511231 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:04.511153    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" event={"ID":"4db4962f-1f52-42d2-94be-91317b48cb8a","Type":"ContainerDied","Data":"b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5"}
Oct 30 22:57:04.511547 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:04.511531    5060 scope.go:122] "RemoveContainer" containerID="b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5"
Oct 30 22:57:04.512589 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:04.512541    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:57:04.512699 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:04.512646    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:57:04.512699 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:04.512691    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:57:04.777086 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:57:04.776994    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:57:05.517312 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:05.517276    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" event={"ID":"4db4962f-1f52-42d2-94be-91317b48cb8a","Type":"ContainerStarted","Data":"43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4"}
Oct 30 22:57:06.371904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:06.371860    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:57:06.372111 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:06.371962    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 22:57:07.371953 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:07.371910    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 22:57:08.372001 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:08.371951    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 22:57:09.371962 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:09.371921    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 22:57:10.387664 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:10.387630    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:57:10.388136 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:10.387956    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:57:10.388767 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:10.388746    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:57:34.775268 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:57:34.775223    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:57:36.316477 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.316445    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"]
Oct 30 22:57:36.318771 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.318753    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.325794 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.325772    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Oct 30 22:57:36.330774 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.330746    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Oct 30 22:57:36.330986 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.330971    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Oct 30 22:57:36.331249 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.331217    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 22:57:36.331317 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.330753    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-qhs6z\""
Oct 30 22:57:36.331317 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.331023    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Oct 30 22:57:36.338803 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.338776    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"]
Oct 30 22:57:36.443228 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.443170    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb2f8986-835a-461c-b800-ea8d9fbb7495-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.443228 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.443232    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.443459 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.443288    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7vd\" (UniqueName: \"kubernetes.io/projected/cb2f8986-835a-461c-b800-ea8d9fbb7495-kube-api-access-gb7vd\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.443459 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.443345    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.443459 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.443389    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.443585 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.443460    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.443585 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.443528    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.443585 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.443561    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2f8986-835a-461c-b800-ea8d9fbb7495-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.544978 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.544920    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.544978 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.544977    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545255 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545013    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2f8986-835a-461c-b800-ea8d9fbb7495-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545255 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545063    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb2f8986-835a-461c-b800-ea8d9fbb7495-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545255 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545099    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545255 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545124    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7vd\" (UniqueName: \"kubernetes.io/projected/cb2f8986-835a-461c-b800-ea8d9fbb7495-kube-api-access-gb7vd\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545255 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545163    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545543 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545433    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545543 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545462    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545543 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545490    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545701 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545551    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545781 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545756    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.545905 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.545889    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb2f8986-835a-461c-b800-ea8d9fbb7495-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.547534 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.547482    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.547661 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.547644    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2f8986-835a-461c-b800-ea8d9fbb7495-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.557904 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.557877    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7vd\" (UniqueName: \"kubernetes.io/projected/cb2f8986-835a-461c-b800-ea8d9fbb7495-kube-api-access-gb7vd\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.633072 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.632993    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:36.917791 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:36.917731    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"]
Oct 30 22:57:36.920074 ip-10-0-138-251 kubenswrapper[5060]: W1030 22:57:36.920040    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2f8986_835a_461c_b800_ea8d9fbb7495.slice/crio-d2a941e07add14a74d22c40b6095a9c0b8a76d689d0d656b296aac85fa0bad94 WatchSource:0}: Error finding container d2a941e07add14a74d22c40b6095a9c0b8a76d689d0d656b296aac85fa0bad94: Status 404 returned error can't find the container with id d2a941e07add14a74d22c40b6095a9c0b8a76d689d0d656b296aac85fa0bad94
Oct 30 22:57:37.681531 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:37.680787    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" event={"ID":"cb2f8986-835a-461c-b800-ea8d9fbb7495","Type":"ContainerStarted","Data":"e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc"}
Oct 30 22:57:37.681531 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:37.680864    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" event={"ID":"cb2f8986-835a-461c-b800-ea8d9fbb7495","Type":"ContainerStarted","Data":"d2a941e07add14a74d22c40b6095a9c0b8a76d689d0d656b296aac85fa0bad94"}
Oct 30 22:57:41.936121 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:41.936032    5060 generic.go:356] "Generic (PLEG): container finished" podID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerID="e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc" exitCode=0
Oct 30 22:57:41.936121 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:41.936096    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" event={"ID":"cb2f8986-835a-461c-b800-ea8d9fbb7495","Type":"ContainerDied","Data":"e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc"}
Oct 30 22:57:42.949253 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:42.949199    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" event={"ID":"cb2f8986-835a-461c-b800-ea8d9fbb7495","Type":"ContainerStarted","Data":"13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e"}
Oct 30 22:57:42.986920 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:42.985250    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" podStartSLOduration=2.453329916 podStartE2EDuration="6.985230616s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.5319007s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:42.982998183 +0000 UTC m=+2343.802736003" watchObservedRunningTime="2026-10-30 22:57:42.985230616 +0000 UTC m=+2343.804968435"
Oct 30 22:57:43.633993 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:43.633954    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:46.633208 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:46.633173    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:57:46.634808 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:46.634779    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 30 22:57:56.633617 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:57:56.633573    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 30 22:58:04.777063 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:58:04.777026    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:58:06.634269 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:06.634223    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 30 22:58:16.544908 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:16.544866    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"]
Oct 30 22:58:16.545479 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:16.545083    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy" containerID="cri-o://43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4" gracePeriod=30
Oct 30 22:58:16.634396 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:16.634345    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 30 22:58:21.804529 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.804440    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:58:21.930326 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930273    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db4962f-1f52-42d2-94be-91317b48cb8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db4962f-1f52-42d2-94be-91317b48cb8a-istiod-ca-cert\") pod \"4db4962f-1f52-42d2-94be-91317b48cb8a\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") "
Oct 30 22:58:21.930711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930360    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-credential-socket\") pod \"4db4962f-1f52-42d2-94be-91317b48cb8a\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") "
Oct 30 22:58:21.930711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930399    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-kube-api-access-jhsz5\" (UniqueName: \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-kube-api-access-jhsz5\") pod \"4db4962f-1f52-42d2-94be-91317b48cb8a\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") "
Oct 30 22:58:21.930711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930429    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-token\" (UniqueName: \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-token\") pod \"4db4962f-1f52-42d2-94be-91317b48cb8a\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") "
Oct 30 22:58:21.930711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930499    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-data\") pod \"4db4962f-1f52-42d2-94be-91317b48cb8a\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") "
Oct 30 22:58:21.930711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930557    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-envoy\") pod \"4db4962f-1f52-42d2-94be-91317b48cb8a\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") "
Oct 30 22:58:21.930711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930610    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-certs\") pod \"4db4962f-1f52-42d2-94be-91317b48cb8a\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") "
Oct 30 22:58:21.930711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930648    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-podinfo\") pod \"4db4962f-1f52-42d2-94be-91317b48cb8a\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") "
Oct 30 22:58:21.930711 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930683    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-socket\") pod \"4db4962f-1f52-42d2-94be-91317b48cb8a\" (UID: \"4db4962f-1f52-42d2-94be-91317b48cb8a\") "
Oct 30 22:58:21.931158 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930723    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db4962f-1f52-42d2-94be-91317b48cb8a-istiod-ca-cert" pod "4db4962f-1f52-42d2-94be-91317b48cb8a" (UID: "4db4962f-1f52-42d2-94be-91317b48cb8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:21.931158 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.930844    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-data" pod "4db4962f-1f52-42d2-94be-91317b48cb8a" (UID: "4db4962f-1f52-42d2-94be-91317b48cb8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:21.931158 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.931021    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-certs" pod "4db4962f-1f52-42d2-94be-91317b48cb8a" (UID: "4db4962f-1f52-42d2-94be-91317b48cb8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:21.931158 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.931098    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-socket" pod "4db4962f-1f52-42d2-94be-91317b48cb8a" (UID: "4db4962f-1f52-42d2-94be-91317b48cb8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:21.931158 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.931129    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db4962f-1f52-42d2-94be-91317b48cb8a-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:21.931419 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.931165    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:21.931419 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.931302    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-credential-socket" pod "4db4962f-1f52-42d2-94be-91317b48cb8a" (UID: "4db4962f-1f52-42d2-94be-91317b48cb8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:21.933137 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.933106    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-podinfo" pod "4db4962f-1f52-42d2-94be-91317b48cb8a" (UID: "4db4962f-1f52-42d2-94be-91317b48cb8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:58:21.933297 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.933273    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-kube-api-access-jhsz5" pod "4db4962f-1f52-42d2-94be-91317b48cb8a" (UID: "4db4962f-1f52-42d2-94be-91317b48cb8a"). InnerVolumeSpecName "kube-api-access-jhsz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:21.933360 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.933333    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-token" pod "4db4962f-1f52-42d2-94be-91317b48cb8a" (UID: "4db4962f-1f52-42d2-94be-91317b48cb8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:21.933416 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:21.933355    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-envoy" pod "4db4962f-1f52-42d2-94be-91317b48cb8a" (UID: "4db4962f-1f52-42d2-94be-91317b48cb8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:22.032808 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.032757    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:22.032808 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.032807    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhsz5\" (UniqueName: \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-kube-api-access-jhsz5\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:22.033015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.032825    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:22.033015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.032840    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:22.033015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.032853    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:22.033015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.032867    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db4962f-1f52-42d2-94be-91317b48cb8a-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:22.033015 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.032880    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db4962f-1f52-42d2-94be-91317b48cb8a-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:22.145205 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.145118    5060 generic.go:356] "Generic (PLEG): container finished" podID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerID="43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4" exitCode=0
Oct 30 22:58:22.145205 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.145191    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"
Oct 30 22:58:22.145393 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.145195    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" event={"ID":"4db4962f-1f52-42d2-94be-91317b48cb8a","Type":"ContainerDied","Data":"43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4"}
Oct 30 22:58:22.145393 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.145299    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb" event={"ID":"4db4962f-1f52-42d2-94be-91317b48cb8a","Type":"ContainerDied","Data":"057ced130b1e0c2b9f524de977c1e109ab03dbd38b8b63876c8a4fec92c09d2e"}
Oct 30 22:58:22.145393 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.145336    5060 scope.go:122] "RemoveContainer" containerID="43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4"
Oct 30 22:58:22.156294 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.156269    5060 scope.go:122] "RemoveContainer" containerID="b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5"
Oct 30 22:58:22.165106 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.165084    5060 scope.go:122] "RemoveContainer" containerID="43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4"
Oct 30 22:58:22.165481 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:58:22.165461    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4\": container with ID starting with 43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4 not found: ID does not exist" containerID="43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4"
Oct 30 22:58:22.165582 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.165489    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4"} err="failed to get container status \"43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4\": rpc error: code = NotFound desc = could not find container \"43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4\": container with ID starting with 43340411b893d86e7c2931e34c07a27217702e0665fd8219e218e556f09311c4 not found: ID does not exist"
Oct 30 22:58:22.165582 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.165527    5060 scope.go:122] "RemoveContainer" containerID="b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5"
Oct 30 22:58:22.165799 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:58:22.165779    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5\": container with ID starting with b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5 not found: ID does not exist" containerID="b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5"
Oct 30 22:58:22.165852 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.165810    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5"} err="failed to get container status \"b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5\": rpc error: code = NotFound desc = could not find container \"b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5\": container with ID starting with b3e81954a6ac41bc331c5d432d6897c107c50b8b52d235460c800ae62df1b9d5 not found: ID does not exist"
Oct 30 22:58:22.169978 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.169954    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"]
Oct 30 22:58:22.175185 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:22.175157    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-64d7d8f984-998gb"]
Oct 30 22:58:23.290359 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:23.290312    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" path="/var/lib/kubelet/pods/4db4962f-1f52-42d2-94be-91317b48cb8a/volumes"
Oct 30 22:58:26.634474 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:26.634422    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 30 22:58:34.775612 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:58:34.775573    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:58:36.634577 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:36.634521    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 30 22:58:39.592081 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:39.592046    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:58:39.596945 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:39.596920    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 22:58:46.634366 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:46.634314    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 30 22:58:56.634392 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:58:56.634347    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Oct 30 22:59:04.775847 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:59:04.775807    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 22:59:06.643082 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:59:06.643047    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:59:06.650561 ip-10-0-138-251 kubenswrapper[5060]: I1030 22:59:06.650537    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 22:59:34.776626 ip-10-0-138-251 kubenswrapper[5060]: E1030 22:59:34.776586    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:00:04.776488 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:00:04.776393    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:00:34.776432 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:00:34.776396    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:01:04.776077 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:01:04.776041    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:01:34.790351 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:01:34.790302    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:01:35.825558 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:35.825520    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"]
Oct 30 23:01:50.848465 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:50.848402    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main" containerID="cri-o://13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e" gracePeriod=45
Oct 30 23:01:51.501433 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.501405    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 23:01:51.597981 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.597883    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2f8986-835a-461c-b800-ea8d9fbb7495-kube-api-access-gb7vd\" (UniqueName: \"kubernetes.io/projected/cb2f8986-835a-461c-b800-ea8d9fbb7495-kube-api-access-gb7vd\") pod \"cb2f8986-835a-461c-b800-ea8d9fbb7495\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") "
Oct 30 23:01:51.597981 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.597929    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb2f8986-835a-461c-b800-ea8d9fbb7495-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2f8986-835a-461c-b800-ea8d9fbb7495-tls-certs\") pod \"cb2f8986-835a-461c-b800-ea8d9fbb7495\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") "
Oct 30 23:01:51.597981 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.597952    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-kserve-provision-location\") pod \"cb2f8986-835a-461c-b800-ea8d9fbb7495\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") "
Oct 30 23:01:51.597981 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.597974    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-home\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-home\") pod \"cb2f8986-835a-461c-b800-ea8d9fbb7495\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") "
Oct 30 23:01:51.598324 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.598038    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-dshm\") pod \"cb2f8986-835a-461c-b800-ea8d9fbb7495\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") "
Oct 30 23:01:51.598324 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.598059    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-model-cache\") pod \"cb2f8986-835a-461c-b800-ea8d9fbb7495\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") "
Oct 30 23:01:51.598324 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.598137    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-tmp-dir\") pod \"cb2f8986-835a-461c-b800-ea8d9fbb7495\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") "
Oct 30 23:01:51.598324 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.598184    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb2f8986-835a-461c-b800-ea8d9fbb7495-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb2f8986-835a-461c-b800-ea8d9fbb7495-cabundle-cert\") pod \"cb2f8986-835a-461c-b800-ea8d9fbb7495\" (UID: \"cb2f8986-835a-461c-b800-ea8d9fbb7495\") "
Oct 30 23:01:51.598616 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.598577    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-home" pod "cb2f8986-835a-461c-b800-ea8d9fbb7495" (UID: "cb2f8986-835a-461c-b800-ea8d9fbb7495"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:51.598691 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.598654    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2f8986-835a-461c-b800-ea8d9fbb7495-cabundle-cert" pod "cb2f8986-835a-461c-b800-ea8d9fbb7495" (UID: "cb2f8986-835a-461c-b800-ea8d9fbb7495"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:51.598854 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.598835    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb2f8986-835a-461c-b800-ea8d9fbb7495-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:51.598919 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.598859    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:51.599058 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.599032    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-model-cache" pod "cb2f8986-835a-461c-b800-ea8d9fbb7495" (UID: "cb2f8986-835a-461c-b800-ea8d9fbb7495"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:51.600403 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.600374    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb2f8986-835a-461c-b800-ea8d9fbb7495-tls-certs" pod "cb2f8986-835a-461c-b800-ea8d9fbb7495" (UID: "cb2f8986-835a-461c-b800-ea8d9fbb7495"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:51.600599 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.600576    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2f8986-835a-461c-b800-ea8d9fbb7495-kube-api-access-gb7vd" pod "cb2f8986-835a-461c-b800-ea8d9fbb7495" (UID: "cb2f8986-835a-461c-b800-ea8d9fbb7495"). InnerVolumeSpecName "kube-api-access-gb7vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:51.600668 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.600598    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-dshm" pod "cb2f8986-835a-461c-b800-ea8d9fbb7495" (UID: "cb2f8986-835a-461c-b800-ea8d9fbb7495"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:51.610913 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.610870    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-tmp-dir" pod "cb2f8986-835a-461c-b800-ea8d9fbb7495" (UID: "cb2f8986-835a-461c-b800-ea8d9fbb7495"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:51.621625 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.621587    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-kserve-provision-location" pod "cb2f8986-835a-461c-b800-ea8d9fbb7495" (UID: "cb2f8986-835a-461c-b800-ea8d9fbb7495"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:01:51.699533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.699475    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7vd\" (UniqueName: \"kubernetes.io/projected/cb2f8986-835a-461c-b800-ea8d9fbb7495-kube-api-access-gb7vd\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:51.699533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.699537    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb2f8986-835a-461c-b800-ea8d9fbb7495-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:51.699738 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.699551    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:51.699738 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.699562    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:51.699738 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.699571    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:51.699738 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:51.699580    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2f8986-835a-461c-b800-ea8d9fbb7495-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:01:52.140528 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.140478    5060 generic.go:356] "Generic (PLEG): container finished" podID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerID="13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e" exitCode=0
Oct 30 23:01:52.140933 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.140561    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" event={"ID":"cb2f8986-835a-461c-b800-ea8d9fbb7495","Type":"ContainerDied","Data":"13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e"}
Oct 30 23:01:52.140933 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.140570    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"
Oct 30 23:01:52.140933 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.140663    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp" event={"ID":"cb2f8986-835a-461c-b800-ea8d9fbb7495","Type":"ContainerDied","Data":"d2a941e07add14a74d22c40b6095a9c0b8a76d689d0d656b296aac85fa0bad94"}
Oct 30 23:01:52.140933 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.140686    5060 scope.go:122] "RemoveContainer" containerID="13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e"
Oct 30 23:01:52.150244 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.150228    5060 scope.go:122] "RemoveContainer" containerID="e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc"
Oct 30 23:01:52.158522 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.158480    5060 scope.go:122] "RemoveContainer" containerID="13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e"
Oct 30 23:01:52.158764 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:01:52.158744    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e\": container with ID starting with 13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e not found: ID does not exist" containerID="13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e"
Oct 30 23:01:52.158815 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.158774    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e"} err="failed to get container status \"13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e\": rpc error: code = NotFound desc = could not find container \"13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e\": container with ID starting with 13b1539b94c6cc9beacfbe1cf0e690c40c5366cc36a4e5fb8eafd0b3d31acd5e not found: ID does not exist"
Oct 30 23:01:52.158815 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.158792    5060 scope.go:122] "RemoveContainer" containerID="e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc"
Oct 30 23:01:52.159025 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:01:52.159004    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc\": container with ID starting with e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc not found: ID does not exist" containerID="e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc"
Oct 30 23:01:52.159067 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.159033    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc"} err="failed to get container status \"e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc\": rpc error: code = NotFound desc = could not find container \"e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc\": container with ID starting with e40b5eeb220dbff123514d629a6a7eb57e3dc5ff8c224e6b0b5dd5606f4ccffc not found: ID does not exist"
Oct 30 23:01:52.167073 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.167051    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"]
Oct 30 23:01:52.171691 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:52.171668    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-685b58654f-d7ctp"]
Oct 30 23:01:53.290107 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:01:53.290075    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" path="/var/lib/kubelet/pods/cb2f8986-835a-461c-b800-ea8d9fbb7495/volumes"
Oct 30 23:02:04.775589 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:02:04.775550    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:02:30.822688 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.822649    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"]
Oct 30 23:02:30.823153 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823037    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy"
Oct 30 23:02:30.823153 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823050    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy"
Oct 30 23:02:30.823153 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823058    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="storage-initializer"
Oct 30 23:02:30.823153 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823063    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="storage-initializer"
Oct 30 23:02:30.823153 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823073    5060 cpu_manager.go:414] "containerMap: removing container" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main"
Oct 30 23:02:30.823153 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823080    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main"
Oct 30 23:02:30.823153 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823095    5060 cpu_manager.go:414] "containerMap: removing container" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy"
Oct 30 23:02:30.823153 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823100    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy"
Oct 30 23:02:30.823395 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823166    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy"
Oct 30 23:02:30.823395 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823175    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2f8986-835a-461c-b800-ea8d9fbb7495" containerName="main"
Oct 30 23:02:30.823395 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.823184    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db4962f-1f52-42d2-94be-91317b48cb8a" containerName="istio-proxy"
Oct 30 23:02:30.826428 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.826410    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:30.840935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.840902    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Oct 30 23:02:30.841082 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.840944    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-bdrmn\""
Oct 30 23:02:30.842286 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.842267    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Oct 30 23:02:30.846306 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.846291    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:02:30.846381 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.846367    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Oct 30 23:02:30.846381 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.846366    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Oct 30 23:02:30.854293 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.854270    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"]
Oct 30 23:02:30.953551 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.953487    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:30.953721 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.953557    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee64280-6ef3-4e41-b555-b674bab53f15-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:30.953721 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.953653    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:30.953721 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.953689    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tplrl\" (UniqueName: \"kubernetes.io/projected/0ee64280-6ef3-4e41-b555-b674bab53f15-kube-api-access-tplrl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:30.953721 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.953720    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:30.953862 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.953795    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:30.953862 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.953837    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:30.953925 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:30.953888    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ee64280-6ef3-4e41-b555-b674bab53f15-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055185 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055147    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055185 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055188    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee64280-6ef3-4e41-b555-b674bab53f15-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055461 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055213    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055461 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055233    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tplrl\" (UniqueName: \"kubernetes.io/projected/0ee64280-6ef3-4e41-b555-b674bab53f15-kube-api-access-tplrl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055461 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055275    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055461 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055296    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055461 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055315    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055461 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055355    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ee64280-6ef3-4e41-b555-b674bab53f15-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055813 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055640    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055813 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055703    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055813 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055723    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.055813 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.055789    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.056245 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.056221    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ee64280-6ef3-4e41-b555-b674bab53f15-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.057725 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.057701    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.058664 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.058644    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee64280-6ef3-4e41-b555-b674bab53f15-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.070697 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.070663    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tplrl\" (UniqueName: \"kubernetes.io/projected/0ee64280-6ef3-4e41-b555-b674bab53f15-kube-api-access-tplrl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.141500 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.141410    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:31.426649 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:31.426083    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"]
Oct 30 23:02:31.428687 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:02:31.428651    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee64280_6ef3_4e41_b555_b674bab53f15.slice/crio-61c2f3f50b12856ae7ce305dcdb8fbcd670416352f50a4ab0961635d64e9837b WatchSource:0}: Error finding container 61c2f3f50b12856ae7ce305dcdb8fbcd670416352f50a4ab0961635d64e9837b: Status 404 returned error can't find the container with id 61c2f3f50b12856ae7ce305dcdb8fbcd670416352f50a4ab0961635d64e9837b
Oct 30 23:02:32.337544 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:32.332751    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" event={"ID":"0ee64280-6ef3-4e41-b555-b674bab53f15","Type":"ContainerStarted","Data":"f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2"}
Oct 30 23:02:32.337544 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:32.332822    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" event={"ID":"0ee64280-6ef3-4e41-b555-b674bab53f15","Type":"ContainerStarted","Data":"61c2f3f50b12856ae7ce305dcdb8fbcd670416352f50a4ab0961635d64e9837b"}
Oct 30 23:02:34.839763 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:02:34.839719    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:02:36.395531 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:36.395041    5060 generic.go:356] "Generic (PLEG): container finished" podID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerID="f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2" exitCode=0
Oct 30 23:02:36.395531 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:36.395173    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" event={"ID":"0ee64280-6ef3-4e41-b555-b674bab53f15","Type":"ContainerDied","Data":"f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2"}
Oct 30 23:02:37.413536 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:37.413476    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" event={"ID":"0ee64280-6ef3-4e41-b555-b674bab53f15","Type":"ContainerStarted","Data":"e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f"}
Oct 30 23:02:37.467711 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:37.467632    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" podStartSLOduration=3.236926412 podStartE2EDuration="7.467611334s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.230684922s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:37.465142635 +0000 UTC m=+2638.284880455" watchObservedRunningTime="2026-10-30 23:02:37.467611334 +0000 UTC m=+2638.287349152"
Oct 30 23:02:38.141676 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:38.141636    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:41.142452 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:41.142417    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:02:41.144140 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:41.144112    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Oct 30 23:02:51.142516 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:02:51.142420    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Oct 30 23:03:01.142043 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:01.141986    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Oct 30 23:03:04.775698 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:03:04.775661    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:03:11.142656 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:11.142610    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Oct 30 23:03:18.598639 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.598598    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"]
Oct 30 23:03:18.602854 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.602826    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:03:18.611527 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.611216    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Oct 30 23:03:18.611527 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.611289    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Oct 30 23:03:18.611527 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.611351    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Oct 30 23:03:18.611527 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.611404    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-84299\""
Oct 30 23:03:18.611527 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.611470    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-ghpb2\""
Oct 30 23:03:18.613621 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.613593    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"]
Oct 30 23:03:18.715764 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.715724    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b47284-63db-4d58-9618-f889f3e8c5d3-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk\" (UID: \"c6b47284-63db-4d58-9618-f889f3e8c5d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:03:18.715937 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.715769    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczwv\" (UniqueName: \"kubernetes.io/projected/c6b47284-63db-4d58-9618-f889f3e8c5d3-kube-api-access-tczwv\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk\" (UID: \"c6b47284-63db-4d58-9618-f889f3e8c5d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:03:18.816295 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.816258    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b47284-63db-4d58-9618-f889f3e8c5d3-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk\" (UID: \"c6b47284-63db-4d58-9618-f889f3e8c5d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:03:18.816473 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.816303    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczwv\" (UniqueName: \"kubernetes.io/projected/c6b47284-63db-4d58-9618-f889f3e8c5d3-kube-api-access-tczwv\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk\" (UID: \"c6b47284-63db-4d58-9618-f889f3e8c5d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:03:18.818833 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.818811    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b47284-63db-4d58-9618-f889f3e8c5d3-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk\" (UID: \"c6b47284-63db-4d58-9618-f889f3e8c5d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:03:18.832919 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.832883    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczwv\" (UniqueName: \"kubernetes.io/projected/c6b47284-63db-4d58-9618-f889f3e8c5d3-kube-api-access-tczwv\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk\" (UID: \"c6b47284-63db-4d58-9618-f889f3e8c5d3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:03:18.921818 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:18.921734    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:03:19.204203 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:19.204177    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"]
Oct 30 23:03:19.206650 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:03:19.206614    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b47284_63db_4d58_9618_f889f3e8c5d3.slice/crio-5d75cbf99c03029388c35ae7ee972fd5c7318e976fe0f2ccc7a4bc9bf8081733 WatchSource:0}: Error finding container 5d75cbf99c03029388c35ae7ee972fd5c7318e976fe0f2ccc7a4bc9bf8081733: Status 404 returned error can't find the container with id 5d75cbf99c03029388c35ae7ee972fd5c7318e976fe0f2ccc7a4bc9bf8081733
Oct 30 23:03:19.637817 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:19.636794    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk" event={"ID":"c6b47284-63db-4d58-9618-f889f3e8c5d3","Type":"ContainerStarted","Data":"9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35"}
Oct 30 23:03:19.637817 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:19.636864    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk" event={"ID":"c6b47284-63db-4d58-9618-f889f3e8c5d3","Type":"ContainerStarted","Data":"5d75cbf99c03029388c35ae7ee972fd5c7318e976fe0f2ccc7a4bc9bf8081733"}
Oct 30 23:03:19.637817 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:19.637707    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:03:19.668536 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:19.667993    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk" podStartSLOduration=1.667971928 podStartE2EDuration="1.667971928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:03:19.662058546 +0000 UTC m=+2680.481796365" watchObservedRunningTime="2026-10-30 23:03:19.667971928 +0000 UTC m=+2680.487709760"
Oct 30 23:03:21.142622 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:21.142574    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Oct 30 23:03:31.142403 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:31.142357    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Oct 30 23:03:34.776628 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:03:34.776594    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:03:39.623564 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:39.623531    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:03:39.630082 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:39.630044    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:03:41.142299 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:41.142252    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Oct 30 23:03:51.152153 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:51.152116    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:03:51.159785 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:51.159759    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:03:51.648972 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:51.648941    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:03:52.634972 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:03:52.634865    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"]
Oct 30 23:04:04.776320 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:04:04.776283    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:04:07.824369 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:07.824304    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main" containerID="cri-o://e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f" gracePeriod=45
Oct 30 23:04:08.153562 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.153416    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Oct 30 23:04:08.398030 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.398000    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:04:08.574940 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.574896    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee64280-6ef3-4e41-b555-b674bab53f15-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ee64280-6ef3-4e41-b555-b674bab53f15-cabundle-cert\") pod \"0ee64280-6ef3-4e41-b555-b674bab53f15\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") "
Oct 30 23:04:08.575130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.574959    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-tmp-dir\") pod \"0ee64280-6ef3-4e41-b555-b674bab53f15\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") "
Oct 30 23:04:08.575130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.574986    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-home\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-home\") pod \"0ee64280-6ef3-4e41-b555-b674bab53f15\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") "
Oct 30 23:04:08.575130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575026    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee64280-6ef3-4e41-b555-b674bab53f15-kube-api-access-tplrl\" (UniqueName: \"kubernetes.io/projected/0ee64280-6ef3-4e41-b555-b674bab53f15-kube-api-access-tplrl\") pod \"0ee64280-6ef3-4e41-b555-b674bab53f15\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") "
Oct 30 23:04:08.575130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575052    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-dshm\") pod \"0ee64280-6ef3-4e41-b555-b674bab53f15\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") "
Oct 30 23:04:08.575130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575083    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ee64280-6ef3-4e41-b555-b674bab53f15-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee64280-6ef3-4e41-b555-b674bab53f15-tls-certs\") pod \"0ee64280-6ef3-4e41-b555-b674bab53f15\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") "
Oct 30 23:04:08.575130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575123    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-kserve-provision-location\") pod \"0ee64280-6ef3-4e41-b555-b674bab53f15\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") "
Oct 30 23:04:08.575459 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575157    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-model-cache\") pod \"0ee64280-6ef3-4e41-b555-b674bab53f15\" (UID: \"0ee64280-6ef3-4e41-b555-b674bab53f15\") "
Oct 30 23:04:08.575459 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575364    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee64280-6ef3-4e41-b555-b674bab53f15-cabundle-cert" pod "0ee64280-6ef3-4e41-b555-b674bab53f15" (UID: "0ee64280-6ef3-4e41-b555-b674bab53f15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:08.575459 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575380    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-tmp-dir" pod "0ee64280-6ef3-4e41-b555-b674bab53f15" (UID: "0ee64280-6ef3-4e41-b555-b674bab53f15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:08.575654 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575555    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-home" pod "0ee64280-6ef3-4e41-b555-b674bab53f15" (UID: "0ee64280-6ef3-4e41-b555-b674bab53f15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:08.575710 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575678    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0ee64280-6ef3-4e41-b555-b674bab53f15-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:08.575710 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575700    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:08.575814 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575716    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:08.575873 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.575841    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-model-cache" pod "0ee64280-6ef3-4e41-b555-b674bab53f15" (UID: "0ee64280-6ef3-4e41-b555-b674bab53f15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:08.577330 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.577302    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-dshm" pod "0ee64280-6ef3-4e41-b555-b674bab53f15" (UID: "0ee64280-6ef3-4e41-b555-b674bab53f15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:08.577432 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.577388    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee64280-6ef3-4e41-b555-b674bab53f15-kube-api-access-tplrl" pod "0ee64280-6ef3-4e41-b555-b674bab53f15" (UID: "0ee64280-6ef3-4e41-b555-b674bab53f15"). InnerVolumeSpecName "kube-api-access-tplrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:08.577718 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.577699    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee64280-6ef3-4e41-b555-b674bab53f15-tls-certs" pod "0ee64280-6ef3-4e41-b555-b674bab53f15" (UID: "0ee64280-6ef3-4e41-b555-b674bab53f15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:08.598260 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.598216    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-kserve-provision-location" pod "0ee64280-6ef3-4e41-b555-b674bab53f15" (UID: "0ee64280-6ef3-4e41-b555-b674bab53f15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:08.676732 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.676689    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tplrl\" (UniqueName: \"kubernetes.io/projected/0ee64280-6ef3-4e41-b555-b674bab53f15-kube-api-access-tplrl\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:08.676732 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.676724    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:08.676732 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.676734    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee64280-6ef3-4e41-b555-b674bab53f15-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:08.676966 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.676743    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:08.676966 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.676754    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee64280-6ef3-4e41-b555-b674bab53f15-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:04:08.873119 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.873028    5060 generic.go:356] "Generic (PLEG): container finished" podID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerID="e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f" exitCode=0
Oct 30 23:04:08.873119 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.873078    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"
Oct 30 23:04:08.873119 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.873107    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" event={"ID":"0ee64280-6ef3-4e41-b555-b674bab53f15","Type":"ContainerDied","Data":"e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f"}
Oct 30 23:04:08.873644 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.873161    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph" event={"ID":"0ee64280-6ef3-4e41-b555-b674bab53f15","Type":"ContainerDied","Data":"61c2f3f50b12856ae7ce305dcdb8fbcd670416352f50a4ab0961635d64e9837b"}
Oct 30 23:04:08.873644 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.873184    5060 scope.go:122] "RemoveContainer" containerID="e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f"
Oct 30 23:04:08.882087 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.882071    5060 scope.go:122] "RemoveContainer" containerID="f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2"
Oct 30 23:04:08.890108 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.890091    5060 scope.go:122] "RemoveContainer" containerID="e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f"
Oct 30 23:04:08.890369 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:04:08.890351    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f\": container with ID starting with e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f not found: ID does not exist" containerID="e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f"
Oct 30 23:04:08.890435 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.890379    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f"} err="failed to get container status \"e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f\": rpc error: code = NotFound desc = could not find container \"e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f\": container with ID starting with e8aa70f5ef5405759b1374a51402df9f29fd90426240bdf1bab7ee6269465e7f not found: ID does not exist"
Oct 30 23:04:08.890435 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.890395    5060 scope.go:122] "RemoveContainer" containerID="f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2"
Oct 30 23:04:08.890669 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:04:08.890649    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2\": container with ID starting with f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2 not found: ID does not exist" containerID="f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2"
Oct 30 23:04:08.890712 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.890677    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2"} err="failed to get container status \"f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2\": rpc error: code = NotFound desc = could not find container \"f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2\": container with ID starting with f4b8f1a70b765b99e660a4fba7c449c7252e5665c8b9630519592f5fd058a8d2 not found: ID does not exist"
Oct 30 23:04:08.900343 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.900322    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"]
Oct 30 23:04:08.907169 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:08.907146    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-555c887948-cxdph"]
Oct 30 23:04:09.289678 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:04:09.289644    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" path="/var/lib/kubelet/pods/0ee64280-6ef3-4e41-b555-b674bab53f15/volumes"
Oct 30 23:04:34.775160 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:04:34.775118    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:05:04.776053 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:05:04.776013    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:05:23.584923 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:05:23.584867    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"]
Oct 30 23:05:31.648945 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:05:31.648862    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk" podUID="c6b47284-63db-4d58-9618-f889f3e8c5d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:34.775130 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:05:34.775088    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:05:38.604589 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:05:38.604537    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk" podUID="c6b47284-63db-4d58-9618-f889f3e8c5d3" containerName="main" containerID="cri-o://9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35" gracePeriod=45
Oct 30 23:05:41.649418 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:05:41.649380    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk" podUID="c6b47284-63db-4d58-9618-f889f3e8c5d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:51.649184 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:05:51.649091    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk" podUID="c6b47284-63db-4d58-9618-f889f3e8c5d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:05:51.649660 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:05:51.649201    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:06:01.648428 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:01.648382    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk" podUID="c6b47284-63db-4d58-9618-f889f3e8c5d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:06:04.775780 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:06:04.775745    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:06:08.855052 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:08.855022    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:06:09.001277 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.001242    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6b47284-63db-4d58-9618-f889f3e8c5d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b47284-63db-4d58-9618-f889f3e8c5d3-tls-certs\") pod \"c6b47284-63db-4d58-9618-f889f3e8c5d3\" (UID: \"c6b47284-63db-4d58-9618-f889f3e8c5d3\") "
Oct 30 23:06:09.001446 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.001285    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b47284-63db-4d58-9618-f889f3e8c5d3-kube-api-access-tczwv\" (UniqueName: \"kubernetes.io/projected/c6b47284-63db-4d58-9618-f889f3e8c5d3-kube-api-access-tczwv\") pod \"c6b47284-63db-4d58-9618-f889f3e8c5d3\" (UID: \"c6b47284-63db-4d58-9618-f889f3e8c5d3\") "
Oct 30 23:06:09.003452 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.003410    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b47284-63db-4d58-9618-f889f3e8c5d3-tls-certs" pod "c6b47284-63db-4d58-9618-f889f3e8c5d3" (UID: "c6b47284-63db-4d58-9618-f889f3e8c5d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:06:09.003586 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.003444    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b47284-63db-4d58-9618-f889f3e8c5d3-kube-api-access-tczwv" pod "c6b47284-63db-4d58-9618-f889f3e8c5d3" (UID: "c6b47284-63db-4d58-9618-f889f3e8c5d3"). InnerVolumeSpecName "kube-api-access-tczwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:09.102376 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.102334    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6b47284-63db-4d58-9618-f889f3e8c5d3-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:06:09.102376 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.102369    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tczwv\" (UniqueName: \"kubernetes.io/projected/c6b47284-63db-4d58-9618-f889f3e8c5d3-kube-api-access-tczwv\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:06:09.400158 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.400124    5060 generic.go:356] "Generic (PLEG): container finished" podID="c6b47284-63db-4d58-9618-f889f3e8c5d3" containerID="9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35" exitCode=0
Oct 30 23:06:09.400158 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.400161    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk" event={"ID":"c6b47284-63db-4d58-9618-f889f3e8c5d3","Type":"ContainerDied","Data":"9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35"}
Oct 30 23:06:09.400380 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.400170    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"
Oct 30 23:06:09.400380 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.400202    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk" event={"ID":"c6b47284-63db-4d58-9618-f889f3e8c5d3","Type":"ContainerDied","Data":"5d75cbf99c03029388c35ae7ee972fd5c7318e976fe0f2ccc7a4bc9bf8081733"}
Oct 30 23:06:09.400380 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.400222    5060 scope.go:122] "RemoveContainer" containerID="9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35"
Oct 30 23:06:09.408655 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.408636    5060 scope.go:122] "RemoveContainer" containerID="9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35"
Oct 30 23:06:09.408951 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:06:09.408929    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35\": container with ID starting with 9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35 not found: ID does not exist" containerID="9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35"
Oct 30 23:06:09.409021 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.408959    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35"} err="failed to get container status \"9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35\": rpc error: code = NotFound desc = could not find container \"9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35\": container with ID starting with 9b648f5b74a114882b49df5c144e1f09d2b4f5d87b516740a8bea6a1c5622d35 not found: ID does not exist"
Oct 30 23:06:09.420260 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.420186    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"]
Oct 30 23:06:09.422154 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:09.422132    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schevq4pk"]
Oct 30 23:06:11.289112 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:06:11.289078    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b47284-63db-4d58-9618-f889f3e8c5d3" path="/var/lib/kubelet/pods/c6b47284-63db-4d58-9618-f889f3e8c5d3/volumes"
Oct 30 23:06:34.775725 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:06:34.775686    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:07:04.775077 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:07:04.775035    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:07:34.775121 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:07:34.775043    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:08:04.775076 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:08:04.775038    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:08:09.108302 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:09.108261    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"]
Oct 30 23:08:09.108896 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:09.108490    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerName="istio-proxy" containerID="cri-o://b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522" gracePeriod=30
Oct 30 23:08:13.814303 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:13.814263    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-xdwvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:08:13.814715 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:13.814333    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:08:14.356856 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.356827    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 23:08:14.387655 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.387568    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e16630-c4bf-4d65-b439-4626bb1a890f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e16630-c4bf-4d65-b439-4626bb1a890f-istiod-ca-cert\") pod \"39e16630-c4bf-4d65-b439-4626bb1a890f\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") "
Oct 30 23:08:14.387655 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.387642    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-credential-socket\") pod \"39e16630-c4bf-4d65-b439-4626bb1a890f\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") "
Oct 30 23:08:14.387861 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.387681    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-podinfo\") pod \"39e16630-c4bf-4d65-b439-4626bb1a890f\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") "
Oct 30 23:08:14.387861 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.387713    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-data\") pod \"39e16630-c4bf-4d65-b439-4626bb1a890f\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") "
Oct 30 23:08:14.387861 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.387753    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-socket\") pod \"39e16630-c4bf-4d65-b439-4626bb1a890f\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") "
Oct 30 23:08:14.387861 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.387797    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-certs\") pod \"39e16630-c4bf-4d65-b439-4626bb1a890f\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") "
Oct 30 23:08:14.387861 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.387831    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-kube-api-access-wrmmh\" (UniqueName: \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-kube-api-access-wrmmh\") pod \"39e16630-c4bf-4d65-b439-4626bb1a890f\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") "
Oct 30 23:08:14.388188 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.387873    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-token\" (UniqueName: \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-token\") pod \"39e16630-c4bf-4d65-b439-4626bb1a890f\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") "
Oct 30 23:08:14.388188 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.388008    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e16630-c4bf-4d65-b439-4626bb1a890f-istiod-ca-cert" pod "39e16630-c4bf-4d65-b439-4626bb1a890f" (UID: "39e16630-c4bf-4d65-b439-4626bb1a890f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:08:14.388188 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.388145    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e16630-c4bf-4d65-b439-4626bb1a890f-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:14.388188 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.388152    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-data" pod "39e16630-c4bf-4d65-b439-4626bb1a890f" (UID: "39e16630-c4bf-4d65-b439-4626bb1a890f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:08:14.388405 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.388213    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-credential-socket" pod "39e16630-c4bf-4d65-b439-4626bb1a890f" (UID: "39e16630-c4bf-4d65-b439-4626bb1a890f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:08:14.388405 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.388307    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-certs" pod "39e16630-c4bf-4d65-b439-4626bb1a890f" (UID: "39e16630-c4bf-4d65-b439-4626bb1a890f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:08:14.388668 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.388644    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-socket" pod "39e16630-c4bf-4d65-b439-4626bb1a890f" (UID: "39e16630-c4bf-4d65-b439-4626bb1a890f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:08:14.390434 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.390413    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-token" pod "39e16630-c4bf-4d65-b439-4626bb1a890f" (UID: "39e16630-c4bf-4d65-b439-4626bb1a890f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:08:14.390535 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.390426    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-kube-api-access-wrmmh" pod "39e16630-c4bf-4d65-b439-4626bb1a890f" (UID: "39e16630-c4bf-4d65-b439-4626bb1a890f"). InnerVolumeSpecName "kube-api-access-wrmmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:08:14.390800 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.390778    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-podinfo" pod "39e16630-c4bf-4d65-b439-4626bb1a890f" (UID: "39e16630-c4bf-4d65-b439-4626bb1a890f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:08:14.489150 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.489108    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-envoy\") pod \"39e16630-c4bf-4d65-b439-4626bb1a890f\" (UID: \"39e16630-c4bf-4d65-b439-4626bb1a890f\") "
Oct 30 23:08:14.489627 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.489595    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:14.489627 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.489626    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:14.489801 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.489641    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:14.489801 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.489654    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:14.489801 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.489667    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:14.489801 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.489680    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrmmh\" (UniqueName: \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-kube-api-access-wrmmh\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:14.489801 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.489693    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:14.491670 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.491640    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-envoy" pod "39e16630-c4bf-4d65-b439-4626bb1a890f" (UID: "39e16630-c4bf-4d65-b439-4626bb1a890f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:08:14.590352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.590297    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e16630-c4bf-4d65-b439-4626bb1a890f-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:08:14.955581 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.955546    5060 generic.go:356] "Generic (PLEG): container finished" podID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerID="b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522" exitCode=0
Oct 30 23:08:14.956069 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.955594    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"
Oct 30 23:08:14.956069 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.955614    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" event={"ID":"39e16630-c4bf-4d65-b439-4626bb1a890f","Type":"ContainerDied","Data":"b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522"}
Oct 30 23:08:14.956069 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.955656    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk" event={"ID":"39e16630-c4bf-4d65-b439-4626bb1a890f","Type":"ContainerDied","Data":"6f7f0f96a3edb6c9180a309f930b29e8d4bbf41dfa8b3b8f2fc6a1deb56b1d42"}
Oct 30 23:08:14.956069 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.955679    5060 scope.go:122] "RemoveContainer" containerID="b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522"
Oct 30 23:08:14.965942 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.965910    5060 scope.go:122] "RemoveContainer" containerID="b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522"
Oct 30 23:08:14.966211 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:08:14.966190    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522\": container with ID starting with b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522 not found: ID does not exist" containerID="b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522"
Oct 30 23:08:14.966260 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.966221    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522"} err="failed to get container status \"b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522\": rpc error: code = NotFound desc = could not find container \"b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522\": container with ID starting with b31c3df25b6e048e7631ee49c6e52602a4168562ab931e50a8efa809bde04522 not found: ID does not exist"
Oct 30 23:08:14.981348 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.981316    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"]
Oct 30 23:08:14.986002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:14.985973    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-xdwvk"]
Oct 30 23:08:15.289381 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:15.289301    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" path="/var/lib/kubelet/pods/39e16630-c4bf-4d65-b439-4626bb1a890f/volumes"
Oct 30 23:08:16.594640 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.594602    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"]
Oct 30 23:08:16.595033 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.594956    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c6b47284-63db-4d58-9618-f889f3e8c5d3" containerName="main"
Oct 30 23:08:16.595033 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.594967    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b47284-63db-4d58-9618-f889f3e8c5d3" containerName="main"
Oct 30 23:08:16.595033 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.594978    5060 cpu_manager.go:414] "containerMap: removing container" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerName="istio-proxy"
Oct 30 23:08:16.595033 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.594983    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerName="istio-proxy"
Oct 30 23:08:16.595033 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.594996    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="storage-initializer"
Oct 30 23:08:16.595033 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.595001    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="storage-initializer"
Oct 30 23:08:16.595033 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.595013    5060 cpu_manager.go:414] "containerMap: removing container" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main"
Oct 30 23:08:16.595033 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.595018    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main"
Oct 30 23:08:16.595278 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.595085    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b47284-63db-4d58-9618-f889f3e8c5d3" containerName="main"
Oct 30 23:08:16.595278 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.595092    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e16630-c4bf-4d65-b439-4626bb1a890f" containerName="istio-proxy"
Oct 30 23:08:16.595278 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.595102    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee64280-6ef3-4e41-b555-b674bab53f15" containerName="main"
Oct 30 23:08:16.599571 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.599549    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.607114 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.607091    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-home\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.607217 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.607125    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kserve-provision-location\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.607217 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.607145    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tmp-dir\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.607305 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.607265    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tls-certs\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.607305 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.607282    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvwkq\" (UniqueName: \"kubernetes.io/projected/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kube-api-access-hvwkq\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.607305 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.607300    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-cabundle-cert\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.607453 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.607412    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-model-cache\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.607496 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.607479    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-dshm\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.613085 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.613062    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Oct 30 23:08:16.613259 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.613140    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:08:16.613259 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.613152    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Oct 30 23:08:16.613259 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.613152    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Oct 30 23:08:16.614288 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.614269    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-8x4bx\""
Oct 30 23:08:16.614373 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.614304    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Oct 30 23:08:16.626650 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.626615    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"]
Oct 30 23:08:16.708248 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708214    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kserve-provision-location\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.708449 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708259    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tmp-dir\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.708449 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708300    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tls-certs\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.708449 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708320    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvwkq\" (UniqueName: \"kubernetes.io/projected/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kube-api-access-hvwkq\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.708449 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708345    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-cabundle-cert\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.708449 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708375    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-model-cache\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.708449 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708419    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-dshm\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.708833 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708458    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-home\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.708833 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708684    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kserve-provision-location\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.708833 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708700    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tmp-dir\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.708833 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708796    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-home\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.709017 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.708995    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-model-cache\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.709298 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.709278    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-cabundle-cert\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.710811 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.710770    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-dshm\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.711540 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.711483    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tls-certs\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.731205 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.731175    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvwkq\" (UniqueName: \"kubernetes.io/projected/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kube-api-access-hvwkq\") pod \"stop-feature-test-kserve-6df697898d-86x8b\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:16.914669 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:16.914562    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:17.261940 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:17.261896    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"]
Oct 30 23:08:17.267794 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:08:17.267768    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebdf56c_87c1_4bda_a28e_83bf97efbcae.slice/crio-2ef59034dc3bb50ab8742824cb71ddb6cd52277e34a17599e45de88537b0d9ac WatchSource:0}: Error finding container 2ef59034dc3bb50ab8742824cb71ddb6cd52277e34a17599e45de88537b0d9ac: Status 404 returned error can't find the container with id 2ef59034dc3bb50ab8742824cb71ddb6cd52277e34a17599e45de88537b0d9ac
Oct 30 23:08:18.002301 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:18.002219    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" event={"ID":"7ebdf56c-87c1-4bda-a28e-83bf97efbcae","Type":"ContainerStarted","Data":"e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e"}
Oct 30 23:08:18.003038 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:18.003006    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" event={"ID":"7ebdf56c-87c1-4bda-a28e-83bf97efbcae","Type":"ContainerStarted","Data":"2ef59034dc3bb50ab8742824cb71ddb6cd52277e34a17599e45de88537b0d9ac"}
Oct 30 23:08:21.021654 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:21.021618    5060 generic.go:356] "Generic (PLEG): container finished" podID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerID="e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e" exitCode=0
Oct 30 23:08:21.022057 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:21.021669    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" event={"ID":"7ebdf56c-87c1-4bda-a28e-83bf97efbcae","Type":"ContainerDied","Data":"e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e"}
Oct 30 23:08:22.034653 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:22.034608    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" event={"ID":"7ebdf56c-87c1-4bda-a28e-83bf97efbcae","Type":"ContainerStarted","Data":"3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18"}
Oct 30 23:08:22.083531 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:22.080075    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podStartSLOduration=3.115162454 podStartE2EDuration="6.080055366s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.964892912s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:08:22.068883902 +0000 UTC m=+2982.888621723" watchObservedRunningTime="2026-10-30 23:08:22.080055366 +0000 UTC m=+2982.899793185"
Oct 30 23:08:22.915235 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:22.915174    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:26.914992 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:26.914956    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:08:26.916619 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:26.916591    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:08:34.776197 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:08:34.776150    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:08:36.915760 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:36.915716    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:08:39.657311 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:39.657283    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:08:39.665382 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:39.665354    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:08:46.915640 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:46.915590    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:08:54.189410 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.189322    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"]
Oct 30 23:08:54.194770 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.194741    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.200112 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.200083    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"]
Oct 30 23:08:54.203980 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.203952    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.208193 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.208163    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"]
Oct 30 23:08:54.220840 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.220796    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"]
Oct 30 23:08:54.298814 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.298771    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.298991 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.298830    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.298991 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.298941    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.298991 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.298987    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.299100 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299008    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.299100 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299035    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb5a636-f3dd-4961-bc4f-da402e02b620-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.299160 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299130    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.299160 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299151    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.299243 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299176    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.299243 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299190    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.299319 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299286    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.299363 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299349    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.299400 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299375    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.299400 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299391    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.299484 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299463    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.299545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299492    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dd4b\" (UniqueName: \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-kube-api-access-5dd4b\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.299545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299538    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqk2q\" (UniqueName: \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-kube-api-access-bqk2q\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.299624 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.299553    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.400990 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.400946    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.400990 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.400987    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.401249 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401010    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.401249 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401041    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.401249 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401071    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.401249 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401094    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.401249 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401137    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.401249 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401160    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dd4b\" (UniqueName: \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-kube-api-access-5dd4b\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.401249 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401187    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqk2q\" (UniqueName: \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-kube-api-access-bqk2q\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.401249 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401211    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.401249 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401238    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.401741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401302    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.401741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401351    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.401741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401373    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.401741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401397    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.401741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401440    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb5a636-f3dd-4961-bc4f-da402e02b620-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.401741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401460    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.401741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401480    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.401741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401501    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.401741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401571    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.402195 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401758    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.402195 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401830    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.402195 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401852    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.402195 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.401917    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.402195 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.402058    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.402447 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.402296    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.402447 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.402312    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.402616 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.402586    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb5a636-f3dd-4961-bc4f-da402e02b620-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.404172 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.404148    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.404172 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.404161    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.404457 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.404437    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.404702 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.404684    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.415062 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.415022    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dd4b\" (UniqueName: \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-kube-api-access-5dd4b\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.415193 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.415074    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqk2q\" (UniqueName: \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-kube-api-access-bqk2q\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.415332 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.415312    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-x9w7j\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.415415 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.415394    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-ghfbj\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.513935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.513897    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:54.525165 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.525133    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:54.799623 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.799594    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"]
Oct 30 23:08:54.801840 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:08:54.801804    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb5a636_f3dd_4961_bc4f_da402e02b620.slice/crio-38dd2340287c49a814a6b2dd40cbfc30ecaf3235dc76a27ffd7a9f1ba88165d3 WatchSource:0}: Error finding container 38dd2340287c49a814a6b2dd40cbfc30ecaf3235dc76a27ffd7a9f1ba88165d3: Status 404 returned error can't find the container with id 38dd2340287c49a814a6b2dd40cbfc30ecaf3235dc76a27ffd7a9f1ba88165d3
Oct 30 23:08:54.803915 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.803879    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:08:54.803998 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.803950    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:08:54.803998 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.803981    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:08:54.809699 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.808225    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"]
Oct 30 23:08:54.815737 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:08:54.814655    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a6f7fa_efe5_4df1_a773_70cf8ca3d0c3.slice/crio-f91edec0c694d105c172f9398e7f7fe4f43f658b45976575c3898c94a21a5314 WatchSource:0}: Error finding container f91edec0c694d105c172f9398e7f7fe4f43f658b45976575c3898c94a21a5314: Status 404 returned error can't find the container with id f91edec0c694d105c172f9398e7f7fe4f43f658b45976575c3898c94a21a5314
Oct 30 23:08:54.818020 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.817978    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:08:54.818123 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.818096    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:08:54.818179 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:54.818132    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:08:55.192487 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.192452    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" event={"ID":"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3","Type":"ContainerStarted","Data":"036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9"}
Oct 30 23:08:55.192994 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.192544    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" event={"ID":"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3","Type":"ContainerStarted","Data":"f91edec0c694d105c172f9398e7f7fe4f43f658b45976575c3898c94a21a5314"}
Oct 30 23:08:55.194389 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.194357    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" event={"ID":"5fb5a636-f3dd-4961-bc4f-da402e02b620","Type":"ContainerStarted","Data":"7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8"}
Oct 30 23:08:55.194545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.194414    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" event={"ID":"5fb5a636-f3dd-4961-bc4f-da402e02b620","Type":"ContainerStarted","Data":"38dd2340287c49a814a6b2dd40cbfc30ecaf3235dc76a27ffd7a9f1ba88165d3"}
Oct 30 23:08:55.220298 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.220221    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" podStartSLOduration=1.220201393 podStartE2EDuration="1.220201393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:08:55.218639956 +0000 UTC m=+3016.038377782" watchObservedRunningTime="2026-10-30 23:08:55.220201393 +0000 UTC m=+3016.039939216"
Oct 30 23:08:55.247685 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.247536    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" podStartSLOduration=1.247491752 podStartE2EDuration="1.247491752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:08:55.243945938 +0000 UTC m=+3016.063683768" watchObservedRunningTime="2026-10-30 23:08:55.247491752 +0000 UTC m=+3016.067229572"
Oct 30 23:08:55.514224 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.514126    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:08:55.515971 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.515942    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghfbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:08:55.516043 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.516000    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 23:08:55.525414 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.525390    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:08:55.526737 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.526709    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-x9w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:08:55.526793 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:55.526765    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 23:08:56.514404 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:56.514365    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghfbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:08:56.514820 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:56.514425    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 23:08:56.525612 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:56.525581    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-x9w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:08:56.525696 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:56.525637    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 23:08:56.915143 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:56.915045    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:08:57.515008 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:57.514966    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghfbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:08:57.515409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:57.515039    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 23:08:57.526273 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:57.526243    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-x9w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:08:57.526418 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:57.526293    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 23:08:58.515003 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:58.514961    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghfbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:08:58.515214 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:58.515194    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 23:08:58.526422 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:58.526392    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-x9w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:08:58.526567 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:58.526438    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 23:08:59.515195 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:59.515158    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghfbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:08:59.515440 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:59.515417    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 23:08:59.525471 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:59.525425    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-x9w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:08:59.525647 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:08:59.525482    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 23:09:00.515269 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:00.515214    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghfbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 23:09:00.515470 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:00.515290    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 23:09:00.525446 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:00.525414    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-x9w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:09:00.525590 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:00.525467    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 23:09:01.518377 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:01.518341    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:09:01.518805 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:01.518753    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:09:01.519434 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:01.519413    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:09:01.529663 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:01.529633    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:09:01.529917 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:01.529898    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:09:01.530695 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:01.530675    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:09:04.776071 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:09:04.776030    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:09:06.915594 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:06.915541    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:09:16.915712 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:16.915664    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:09:26.915565 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:26.915492    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:09:34.777265 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:09:34.777228    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:09:36.916011 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:36.915963    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:09:46.916115 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:46.916061    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:09:56.915461 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:09:56.915414    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:10:04.776893 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:10:04.776849    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:10:06.914953 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:06.914906    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:10:16.924958 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:16.924909    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:10:16.932982 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:16.932951    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:10:18.221922 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:18.221883    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"]
Oct 30 23:10:26.825732 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.825636    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"]
Oct 30 23:10:26.828156 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.828134    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.852906 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.852870    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"]
Oct 30 23:10:26.895610 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.895565    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-dshm\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.895792 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.895664    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-tmp-dir\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.895792 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.895723    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-home\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.895792 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.895757    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-model-cache\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.895792 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.895786    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xwk5\" (UniqueName: \"kubernetes.io/projected/2dd49a4b-9f09-4f9c-8574-18406edda372-kube-api-access-5xwk5\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.895983 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.895908    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-kserve-provision-location\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.895983 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.895943    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd49a4b-9f09-4f9c-8574-18406edda372-tls-certs\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.895983 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.895973    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd49a4b-9f09-4f9c-8574-18406edda372-cabundle-cert\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.996688 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.996651    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-home\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.996879 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.996709    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-model-cache\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.996879 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.996755    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xwk5\" (UniqueName: \"kubernetes.io/projected/2dd49a4b-9f09-4f9c-8574-18406edda372-kube-api-access-5xwk5\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.996879 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.996802    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-kserve-provision-location\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.996879 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.996833    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd49a4b-9f09-4f9c-8574-18406edda372-tls-certs\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.996879 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.996864    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd49a4b-9f09-4f9c-8574-18406edda372-cabundle-cert\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.997130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.996926    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-dshm\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.997130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.996966    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-tmp-dir\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.997230 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.997180    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-home\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.997230 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.997193    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-model-cache\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.997230 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.997213    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-kserve-provision-location\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.997354 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.997250    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-tmp-dir\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.997851 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.997826    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd49a4b-9f09-4f9c-8574-18406edda372-cabundle-cert\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:26.999643 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:26.999612    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-dshm\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:27.000028 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:27.000009    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd49a4b-9f09-4f9c-8574-18406edda372-tls-certs\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:27.013993 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:27.013958    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xwk5\" (UniqueName: \"kubernetes.io/projected/2dd49a4b-9f09-4f9c-8574-18406edda372-kube-api-access-5xwk5\") pod \"stop-feature-test-kserve-6df697898d-jn56p\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:27.140257 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:27.140159    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:27.426719 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:27.426633    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"]
Oct 30 23:10:27.430228 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:10:27.430198    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd49a4b_9f09_4f9c_8574_18406edda372.slice/crio-2d274bd9992ebbe2fdd36c3b0e64dc1ca643eb1ea1337449f11c058af4e9a7c9 WatchSource:0}: Error finding container 2d274bd9992ebbe2fdd36c3b0e64dc1ca643eb1ea1337449f11c058af4e9a7c9: Status 404 returned error can't find the container with id 2d274bd9992ebbe2fdd36c3b0e64dc1ca643eb1ea1337449f11c058af4e9a7c9
Oct 30 23:10:27.677844 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:27.677734    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" event={"ID":"2dd49a4b-9f09-4f9c-8574-18406edda372","Type":"ContainerStarted","Data":"9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24"}
Oct 30 23:10:27.677844 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:27.677807    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" event={"ID":"2dd49a4b-9f09-4f9c-8574-18406edda372","Type":"ContainerStarted","Data":"2d274bd9992ebbe2fdd36c3b0e64dc1ca643eb1ea1337449f11c058af4e9a7c9"}
Oct 30 23:10:30.693908 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:30.693873    5060 generic.go:356] "Generic (PLEG): container finished" podID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerID="9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24" exitCode=0
Oct 30 23:10:30.694454 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:30.693953    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" event={"ID":"2dd49a4b-9f09-4f9c-8574-18406edda372","Type":"ContainerDied","Data":"9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24"}
Oct 30 23:10:31.700640 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:31.700597    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" event={"ID":"2dd49a4b-9f09-4f9c-8574-18406edda372","Type":"ContainerStarted","Data":"1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef"}
Oct 30 23:10:31.732392 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:31.732337    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podStartSLOduration=3.401725144 podStartE2EDuration="5.732322355s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.330597211s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:10:31.729798279 +0000 UTC m=+3112.549536133" watchObservedRunningTime="2026-10-30 23:10:31.732322355 +0000 UTC m=+3112.552060174"
Oct 30 23:10:32.140750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:32.140661    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:33.648825 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:33.648779    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" containerID="cri-o://3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18" gracePeriod=45
Oct 30 23:10:33.926113 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:33.926007    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 30 23:10:34.525793 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.525762    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:10:34.686705 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.686665    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-dshm\") pod \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") "
Oct 30 23:10:34.687178 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.686753    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-home\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-home\") pod \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") "
Oct 30 23:10:34.687178 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.686773    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kube-api-access-hvwkq\" (UniqueName: \"kubernetes.io/projected/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kube-api-access-hvwkq\") pod \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") "
Oct 30 23:10:34.687178 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.686799    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tmp-dir\") pod \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") "
Oct 30 23:10:34.687178 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.686817    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-model-cache\") pod \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") "
Oct 30 23:10:34.687178 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.686882    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-cabundle-cert\") pod \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") "
Oct 30 23:10:34.687178 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.686916    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tls-certs\") pod \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") "
Oct 30 23:10:34.687178 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.686959    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kserve-provision-location\") pod \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\" (UID: \"7ebdf56c-87c1-4bda-a28e-83bf97efbcae\") "
Oct 30 23:10:34.687557 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.687365    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-cabundle-cert" pod "7ebdf56c-87c1-4bda-a28e-83bf97efbcae" (UID: "7ebdf56c-87c1-4bda-a28e-83bf97efbcae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:10:34.687940 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.687909    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-home" pod "7ebdf56c-87c1-4bda-a28e-83bf97efbcae" (UID: "7ebdf56c-87c1-4bda-a28e-83bf97efbcae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:10:34.688300 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.688267    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-model-cache" pod "7ebdf56c-87c1-4bda-a28e-83bf97efbcae" (UID: "7ebdf56c-87c1-4bda-a28e-83bf97efbcae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:10:34.689066 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.689035    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-dshm" pod "7ebdf56c-87c1-4bda-a28e-83bf97efbcae" (UID: "7ebdf56c-87c1-4bda-a28e-83bf97efbcae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:10:34.689404 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.689377    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kube-api-access-hvwkq" pod "7ebdf56c-87c1-4bda-a28e-83bf97efbcae" (UID: "7ebdf56c-87c1-4bda-a28e-83bf97efbcae"). InnerVolumeSpecName "kube-api-access-hvwkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:10:34.690237 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.690204    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tls-certs" pod "7ebdf56c-87c1-4bda-a28e-83bf97efbcae" (UID: "7ebdf56c-87c1-4bda-a28e-83bf97efbcae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:10:34.696783 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.696745    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tmp-dir" pod "7ebdf56c-87c1-4bda-a28e-83bf97efbcae" (UID: "7ebdf56c-87c1-4bda-a28e-83bf97efbcae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:10:34.709899 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.709857    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kserve-provision-location" pod "7ebdf56c-87c1-4bda-a28e-83bf97efbcae" (UID: "7ebdf56c-87c1-4bda-a28e-83bf97efbcae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:10:34.744062 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.744009    5060 generic.go:356] "Generic (PLEG): container finished" podID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerID="3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18" exitCode=0
Oct 30 23:10:34.744236 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.744062    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" event={"ID":"7ebdf56c-87c1-4bda-a28e-83bf97efbcae","Type":"ContainerDied","Data":"3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18"}
Oct 30 23:10:34.744236 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.744103    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"
Oct 30 23:10:34.744236 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.744126    5060 scope.go:122] "RemoveContainer" containerID="3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18"
Oct 30 23:10:34.744236 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.744116    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b" event={"ID":"7ebdf56c-87c1-4bda-a28e-83bf97efbcae","Type":"ContainerDied","Data":"2ef59034dc3bb50ab8742824cb71ddb6cd52277e34a17599e45de88537b0d9ac"}
Oct 30 23:10:34.756674 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.756652    5060 scope.go:122] "RemoveContainer" containerID="e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e"
Oct 30 23:10:34.766383 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.766359    5060 scope.go:122] "RemoveContainer" containerID="3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18"
Oct 30 23:10:34.766725 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:10:34.766696    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18\": container with ID starting with 3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18 not found: ID does not exist" containerID="3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18"
Oct 30 23:10:34.766844 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.766741    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18"} err="failed to get container status \"3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18\": rpc error: code = NotFound desc = could not find container \"3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18\": container with ID starting with 3057841c5a126ebd2f93bcc10d386a8d6811c1a589851b443885180def736b18 not found: ID does not exist"
Oct 30 23:10:34.766844 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.766766    5060 scope.go:122] "RemoveContainer" containerID="e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e"
Oct 30 23:10:34.767064 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:10:34.767044    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e\": container with ID starting with e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e not found: ID does not exist" containerID="e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e"
Oct 30 23:10:34.767113 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.767074    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e"} err="failed to get container status \"e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e\": rpc error: code = NotFound desc = could not find container \"e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e\": container with ID starting with e659f2e401f13eba3aa4dc545d0528fa8c91cb3f1229ed62adea65174c05a32e not found: ID does not exist"
Oct 30 23:10:34.777442 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.777410    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"]
Oct 30 23:10:34.778968 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:10:34.778940    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:10:34.782672 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.782648    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-86x8b"]
Oct 30 23:10:34.788051 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.788032    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:34.788114 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.788054    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvwkq\" (UniqueName: \"kubernetes.io/projected/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kube-api-access-hvwkq\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:34.788114 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.788067    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:34.788114 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.788076    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:34.788114 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.788084    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:34.788114 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.788092    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:34.788114 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.788100    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:34.788114 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:34.788109    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebdf56c-87c1-4bda-a28e-83bf97efbcae-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:10:35.290761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:35.290717    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" path="/var/lib/kubelet/pods/7ebdf56c-87c1-4bda-a28e-83bf97efbcae/volumes"
Oct 30 23:10:37.140599 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:37.140556    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:10:37.142213 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:37.142177    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:10:47.141569 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:47.141493    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:10:57.140982 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:10:57.140925    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:11:04.776711 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:11:04.776669    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:11:07.141016 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:11:07.140960    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:11:17.140680 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:11:17.140629    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:11:27.140833 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:11:27.140784    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:11:34.777590 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:11:34.777544    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:11:37.141191 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:11:37.141147    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:11:47.140629 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:11:47.140575    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:11:57.141459 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:11:57.141364    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:12:04.776082 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:12:04.776038    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:12:07.140939 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:07.140889    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:12:17.141095 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:17.141040    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:12:27.151052 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:27.151014    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:12:27.159653 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:27.159621    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:12:28.477142 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:28.477106    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"]
Oct 30 23:12:34.775396 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:12:34.775288    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:12:43.500141 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:43.500099    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" containerID="cri-o://1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef" gracePeriod=45
Oct 30 23:12:44.151799 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.151754    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Oct 30 23:12:44.250135 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.250099    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:12:44.387415 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.387311    5060 generic.go:356] "Generic (PLEG): container finished" podID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerID="1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef" exitCode=0
Oct 30 23:12:44.387415 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.387358    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" event={"ID":"2dd49a4b-9f09-4f9c-8574-18406edda372","Type":"ContainerDied","Data":"1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef"}
Oct 30 23:12:44.387415 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.387415    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p" event={"ID":"2dd49a4b-9f09-4f9c-8574-18406edda372","Type":"ContainerDied","Data":"2d274bd9992ebbe2fdd36c3b0e64dc1ca643eb1ea1337449f11c058af4e9a7c9"}
Oct 30 23:12:44.387750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.387436    5060 scope.go:122] "RemoveContainer" containerID="1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef"
Oct 30 23:12:44.387750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.387373    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"
Oct 30 23:12:44.396014 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.395993    5060 scope.go:122] "RemoveContainer" containerID="9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24"
Oct 30 23:12:44.404096 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.404078    5060 scope.go:122] "RemoveContainer" containerID="1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef"
Oct 30 23:12:44.404347 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:12:44.404328    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef\": container with ID starting with 1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef not found: ID does not exist" containerID="1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef"
Oct 30 23:12:44.404392 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.404357    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef"} err="failed to get container status \"1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef\": rpc error: code = NotFound desc = could not find container \"1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef\": container with ID starting with 1ff19212866a40a395f7434161fe88ddcb5f3cdd5f47466201e8d992acda18ef not found: ID does not exist"
Oct 30 23:12:44.404392 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.404376    5060 scope.go:122] "RemoveContainer" containerID="9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24"
Oct 30 23:12:44.404635 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:12:44.404615    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24\": container with ID starting with 9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24 not found: ID does not exist" containerID="9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24"
Oct 30 23:12:44.404702 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.404645    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24"} err="failed to get container status \"9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24\": rpc error: code = NotFound desc = could not find container \"9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24\": container with ID starting with 9137b30c8219a00e526da278726f2348131d78fa5b798b7138579189a27e7e24 not found: ID does not exist"
Oct 30 23:12:44.426609 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.426573    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-kserve-provision-location\") pod \"2dd49a4b-9f09-4f9c-8574-18406edda372\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") "
Oct 30 23:12:44.426759 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.426636    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-tmp-dir\") pod \"2dd49a4b-9f09-4f9c-8574-18406edda372\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") "
Oct 30 23:12:44.426759 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.426745    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-model-cache\") pod \"2dd49a4b-9f09-4f9c-8574-18406edda372\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") "
Oct 30 23:12:44.426867 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.426776    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd49a4b-9f09-4f9c-8574-18406edda372-kube-api-access-5xwk5\" (UniqueName: \"kubernetes.io/projected/2dd49a4b-9f09-4f9c-8574-18406edda372-kube-api-access-5xwk5\") pod \"2dd49a4b-9f09-4f9c-8574-18406edda372\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") "
Oct 30 23:12:44.426867 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.426808    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd49a4b-9f09-4f9c-8574-18406edda372-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd49a4b-9f09-4f9c-8574-18406edda372-tls-certs\") pod \"2dd49a4b-9f09-4f9c-8574-18406edda372\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") "
Oct 30 23:12:44.426867 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.426855    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-dshm\") pod \"2dd49a4b-9f09-4f9c-8574-18406edda372\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") "
Oct 30 23:12:44.427010 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.426885    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-home\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-home\") pod \"2dd49a4b-9f09-4f9c-8574-18406edda372\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") "
Oct 30 23:12:44.427010 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.426948    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd49a4b-9f09-4f9c-8574-18406edda372-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd49a4b-9f09-4f9c-8574-18406edda372-cabundle-cert\") pod \"2dd49a4b-9f09-4f9c-8574-18406edda372\" (UID: \"2dd49a4b-9f09-4f9c-8574-18406edda372\") "
Oct 30 23:12:44.427102 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.427062    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-model-cache" pod "2dd49a4b-9f09-4f9c-8574-18406edda372" (UID: "2dd49a4b-9f09-4f9c-8574-18406edda372"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:12:44.427335 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.427260    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:12:44.427546 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.427523    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd49a4b-9f09-4f9c-8574-18406edda372-cabundle-cert" pod "2dd49a4b-9f09-4f9c-8574-18406edda372" (UID: "2dd49a4b-9f09-4f9c-8574-18406edda372"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:12:44.427946 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.427916    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-home" pod "2dd49a4b-9f09-4f9c-8574-18406edda372" (UID: "2dd49a4b-9f09-4f9c-8574-18406edda372"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:12:44.429085 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.429052    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-dshm" pod "2dd49a4b-9f09-4f9c-8574-18406edda372" (UID: "2dd49a4b-9f09-4f9c-8574-18406edda372"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:12:44.429166 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.429122    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd49a4b-9f09-4f9c-8574-18406edda372-tls-certs" pod "2dd49a4b-9f09-4f9c-8574-18406edda372" (UID: "2dd49a4b-9f09-4f9c-8574-18406edda372"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:12:44.429282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.429266    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd49a4b-9f09-4f9c-8574-18406edda372-kube-api-access-5xwk5" pod "2dd49a4b-9f09-4f9c-8574-18406edda372" (UID: "2dd49a4b-9f09-4f9c-8574-18406edda372"). InnerVolumeSpecName "kube-api-access-5xwk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:12:44.444682 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.444653    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-tmp-dir" pod "2dd49a4b-9f09-4f9c-8574-18406edda372" (UID: "2dd49a4b-9f09-4f9c-8574-18406edda372"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:12:44.452374 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.452344    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-kserve-provision-location" pod "2dd49a4b-9f09-4f9c-8574-18406edda372" (UID: "2dd49a4b-9f09-4f9c-8574-18406edda372"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:12:44.528547 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.528490    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xwk5\" (UniqueName: \"kubernetes.io/projected/2dd49a4b-9f09-4f9c-8574-18406edda372-kube-api-access-5xwk5\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:12:44.528547 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.528548    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd49a4b-9f09-4f9c-8574-18406edda372-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:12:44.528964 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.528558    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:12:44.528964 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.528568    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:12:44.528964 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.528577    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd49a4b-9f09-4f9c-8574-18406edda372-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:12:44.528964 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.528586    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:12:44.528964 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.528594    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd49a4b-9f09-4f9c-8574-18406edda372-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:12:44.713181 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.713149    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"]
Oct 30 23:12:44.718141 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:44.718107    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6df697898d-jn56p"]
Oct 30 23:12:45.289278 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:12:45.289242    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" path="/var/lib/kubelet/pods/2dd49a4b-9f09-4f9c-8574-18406edda372/volumes"
Oct 30 23:13:04.775153 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:13:04.775114    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:13:23.118324 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118230    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"]
Oct 30 23:13:23.118784 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118684    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="storage-initializer"
Oct 30 23:13:23.118784 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118697    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="storage-initializer"
Oct 30 23:13:23.118784 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118724    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main"
Oct 30 23:13:23.118784 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118729    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main"
Oct 30 23:13:23.118784 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118737    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="storage-initializer"
Oct 30 23:13:23.118784 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118743    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="storage-initializer"
Oct 30 23:13:23.118784 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118753    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main"
Oct 30 23:13:23.118784 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118758    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main"
Oct 30 23:13:23.119035 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118836    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebdf56c-87c1-4bda-a28e-83bf97efbcae" containerName="main"
Oct 30 23:13:23.119035 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.118849    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd49a4b-9f09-4f9c-8574-18406edda372" containerName="main"
Oct 30 23:13:23.122144 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.122125    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.132045 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.132013    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:13:23.132166 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.132052    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Oct 30 23:13:23.132166 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.132058    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Oct 30 23:13:23.132766 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.132746    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-jbs9j\""
Oct 30 23:13:23.132871 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.132860    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Oct 30 23:13:23.132929 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.132885    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Oct 30 23:13:23.138316 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.138287    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"]
Oct 30 23:13:23.255953 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.255899    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.256148 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.255973    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.256148 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.256049    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-dshm\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.256148 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.256086    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tmp-dir\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.256148 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.256110    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tls-certs\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.256355 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.256195    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-home\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.256355 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.256226    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlwps\" (UniqueName: \"kubernetes.io/projected/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kube-api-access-zlwps\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.256355 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.256324    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-model-cache\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.357328 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357283    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.357328 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357330    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-dshm\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.357642 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357351    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tmp-dir\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.357642 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357403    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tls-certs\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.357642 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357444    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-home\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.357642 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357473    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlwps\" (UniqueName: \"kubernetes.io/projected/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kube-api-access-zlwps\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.357642 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357533    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-model-cache\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.357642 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357606    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.357950 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357810    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.357950 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357855    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tmp-dir\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.358043 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.357964    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-model-cache\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.358272 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.358247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-home\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.358489 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.358469    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-cabundle-cert\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.359829 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.359806    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-dshm\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.360225 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.360204    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tls-certs\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.390268 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.390185    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlwps\" (UniqueName: \"kubernetes.io/projected/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kube-api-access-zlwps\") pod \"lora-single-adapter-test-kserve-7d79587f48-cng2s\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.438200 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.438157    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:23.722268 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:23.722235    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"]
Oct 30 23:13:23.724228 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:13:23.724197    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1479a96_d4ac_46cc_a8f9_ac1b1d31fe40.slice/crio-59ab28cb5592ec51720ebe736859f00e2d55e9ccd41a70acfc498bce040c5aca WatchSource:0}: Error finding container 59ab28cb5592ec51720ebe736859f00e2d55e9ccd41a70acfc498bce040c5aca: Status 404 returned error can't find the container with id 59ab28cb5592ec51720ebe736859f00e2d55e9ccd41a70acfc498bce040c5aca
Oct 30 23:13:24.587853 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:24.586927    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" event={"ID":"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40","Type":"ContainerStarted","Data":"b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238"}
Oct 30 23:13:24.587853 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:24.587002    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" event={"ID":"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40","Type":"ContainerStarted","Data":"59ab28cb5592ec51720ebe736859f00e2d55e9ccd41a70acfc498bce040c5aca"}
Oct 30 23:13:28.755545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:28.755218    5060 generic.go:356] "Generic (PLEG): container finished" podID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerID="b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238" exitCode=0
Oct 30 23:13:28.755545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:28.755408    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" event={"ID":"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40","Type":"ContainerDied","Data":"b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238"}
Oct 30 23:13:29.777215 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:29.777174    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" event={"ID":"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40","Type":"ContainerStarted","Data":"367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be"}
Oct 30 23:13:29.822713 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:29.822631    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podStartSLOduration=2.026795686 podStartE2EDuration="6.822608772s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.795813086s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:29.81722494 +0000 UTC m=+3290.636962763" watchObservedRunningTime="2026-10-30 23:13:29.822608772 +0000 UTC m=+3290.642346592"
Oct 30 23:13:30.438709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:30.438613    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:33.439018 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:33.438973    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:13:33.440638 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:33.440605    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 23:13:34.778104 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:13:34.778062    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:13:39.688695 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:39.688660    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:13:39.697957 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:39.697917    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:13:43.438961 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:43.438912    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 23:13:53.439351 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:13:53.439305    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 23:14:03.438845 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:03.438792    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 23:14:04.776943 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:14:04.776906    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:14:13.438620 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:13.438576    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 23:14:23.438945 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:23.438898    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 23:14:33.439684 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:33.439639    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 23:14:34.794158 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:14:34.794103    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:14:43.438810 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:43.438767    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 23:14:45.755978 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.755935    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"]
Oct 30 23:14:45.759654 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.759629    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.780754 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.780723    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"]
Oct 30 23:14:45.783113 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.783076    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Oct 30 23:14:45.783258 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.783213    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Oct 30 23:14:45.783258 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.783220    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-pjngx\""
Oct 30 23:14:45.824674 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.824645    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:14:45.824674 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.824645    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Oct 30 23:14:45.824674 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.824645    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Oct 30 23:14:45.875683 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.875643    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.875683 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.875688    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.875883 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.875735    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.875883 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.875812    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.875883 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.875864    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.876012 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.875888    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.876012 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.875930    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr4n8\" (UniqueName: \"kubernetes.io/projected/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kube-api-access-vr4n8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.876117 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.876043    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.977554 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.977481    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.977761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.977571    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.977761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.977595    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.977761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.977629    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.977761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.977657    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.977761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.977711    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr4n8\" (UniqueName: \"kubernetes.io/projected/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kube-api-access-vr4n8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.977761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.977744    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.978070 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.977794    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.978070 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.977993    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.978070 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.978027    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.978293 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.978260    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.978410 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.978293    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.978410 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.978318    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.980120 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.980092    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.980411 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.980393    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:45.994607 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:45.994572    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr4n8\" (UniqueName: \"kubernetes.io/projected/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kube-api-access-vr4n8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:46.076039 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:46.075942    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:46.365832 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:46.365800    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"]
Oct 30 23:14:46.367579 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:14:46.367547    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ed9f19_6e5e_46f5_8d7a_c318c4c405a3.slice/crio-20c402a6ad6af2d94e613a12d3e44feebd86d73f067e429b7e00fdef6ca20481 WatchSource:0}: Error finding container 20c402a6ad6af2d94e613a12d3e44feebd86d73f067e429b7e00fdef6ca20481: Status 404 returned error can't find the container with id 20c402a6ad6af2d94e613a12d3e44feebd86d73f067e429b7e00fdef6ca20481
Oct 30 23:14:47.167856 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:47.167810    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" event={"ID":"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3","Type":"ContainerStarted","Data":"a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81"}
Oct 30 23:14:47.168372 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:47.167869    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" event={"ID":"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3","Type":"ContainerStarted","Data":"20c402a6ad6af2d94e613a12d3e44feebd86d73f067e429b7e00fdef6ca20481"}
Oct 30 23:14:49.183595 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:49.183558    5060 generic.go:356] "Generic (PLEG): container finished" podID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerID="a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81" exitCode=0
Oct 30 23:14:49.183973 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:49.183633    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" event={"ID":"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3","Type":"ContainerDied","Data":"a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81"}
Oct 30 23:14:50.192046 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:50.191993    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" event={"ID":"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3","Type":"ContainerStarted","Data":"9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3"}
Oct 30 23:14:50.223269 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:50.223196    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podStartSLOduration=2.703303468 podStartE2EDuration="5.223180957s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.519877489s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:14:50.220403585 +0000 UTC m=+3371.040141404" watchObservedRunningTime="2026-10-30 23:14:50.223180957 +0000 UTC m=+3371.042918775"
Oct 30 23:14:53.448669 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:53.448638    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:14:53.456639 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:53.456606    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:14:56.077137 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:56.077096    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:56.077630 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:56.077207    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:14:56.078858 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:14:56.078829    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:15:04.776975 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:15:04.776928    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:15:06.076930 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:06.076870    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:15:16.077019 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:16.076957    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:15:19.601797 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:19.601765    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7d79587f48-cng2s_c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40/storage-initializer/0.log"
Oct 30 23:15:19.627476 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:19.627435    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-7d79587f48-cng2s_c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40/main/0.log"
Oct 30 23:15:24.316927 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:24.316889    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"]
Oct 30 23:15:26.077015 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:26.076970    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:15:34.777051 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:15:34.777009    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:15:36.077286 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:36.077243    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:15:39.339702 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:39.339651    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main" containerID="cri-o://367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be" gracePeriod=45
Oct 30 23:15:39.449274 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:39.449226    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 23:15:40.006944 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.006913    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:15:40.122259 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122163    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-cabundle-cert\") pod \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") "
Oct 30 23:15:40.122259 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122233    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-home\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-home\") pod \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") "
Oct 30 23:15:40.122496 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122275    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kserve-provision-location\") pod \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") "
Oct 30 23:15:40.122496 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122301    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-model-cache\") pod \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") "
Oct 30 23:15:40.122496 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122336    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kube-api-access-zlwps\" (UniqueName: \"kubernetes.io/projected/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kube-api-access-zlwps\") pod \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") "
Oct 30 23:15:40.122496 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122380    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tmp-dir\") pod \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") "
Oct 30 23:15:40.122496 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122406    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tls-certs\") pod \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") "
Oct 30 23:15:40.122496 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122475    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-dshm\") pod \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\" (UID: \"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40\") "
Oct 30 23:15:40.122827 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122643    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-model-cache" pod "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" (UID: "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:15:40.122827 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122670    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-cabundle-cert" pod "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" (UID: "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:15:40.122827 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122702    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-home" pod "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" (UID: "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:15:40.122981 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122907    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:15:40.123078 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.122944    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:15:40.123133 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.123083    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:15:40.124990 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.124968    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-dshm" pod "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" (UID: "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:15:40.125755 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.125733    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tls-certs" pod "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" (UID: "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:15:40.125844 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.125732    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kube-api-access-zlwps" pod "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" (UID: "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40"). InnerVolumeSpecName "kube-api-access-zlwps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:15:40.133865 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.133835    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tmp-dir" pod "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" (UID: "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:15:40.141852 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.141809    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kserve-provision-location" pod "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" (UID: "c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:15:40.224236 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.224195    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:15:40.224236 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.224230    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:15:40.224236 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.224240    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlwps\" (UniqueName: \"kubernetes.io/projected/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-kube-api-access-zlwps\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:15:40.224479 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.224250    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:15:40.224479 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.224260    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:15:40.456684 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.456647    5060 generic.go:356] "Generic (PLEG): container finished" podID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerID="367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be" exitCode=0
Oct 30 23:15:40.457144 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.456696    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"
Oct 30 23:15:40.457144 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.456738    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" event={"ID":"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40","Type":"ContainerDied","Data":"367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be"}
Oct 30 23:15:40.457144 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.456809    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s" event={"ID":"c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40","Type":"ContainerDied","Data":"59ab28cb5592ec51720ebe736859f00e2d55e9ccd41a70acfc498bce040c5aca"}
Oct 30 23:15:40.457144 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.456843    5060 scope.go:122] "RemoveContainer" containerID="367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be"
Oct 30 23:15:40.466468 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.466443    5060 scope.go:122] "RemoveContainer" containerID="b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238"
Oct 30 23:15:40.474890 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.474864    5060 scope.go:122] "RemoveContainer" containerID="367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be"
Oct 30 23:15:40.475143 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:15:40.475122    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be\": container with ID starting with 367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be not found: ID does not exist" containerID="367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be"
Oct 30 23:15:40.475210 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.475161    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be"} err="failed to get container status \"367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be\": rpc error: code = NotFound desc = could not find container \"367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be\": container with ID starting with 367df23e848faa1552ce2e206f3ddb4e2c417e968f9618eb69b50132620973be not found: ID does not exist"
Oct 30 23:15:40.475210 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.475186    5060 scope.go:122] "RemoveContainer" containerID="b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238"
Oct 30 23:15:40.475462 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:15:40.475445    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238\": container with ID starting with b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238 not found: ID does not exist" containerID="b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238"
Oct 30 23:15:40.475537 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.475468    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238"} err="failed to get container status \"b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238\": rpc error: code = NotFound desc = could not find container \"b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238\": container with ID starting with b4fed888270054599fe35719fec5bbff124d5e09560602ffbaae9ef9c9de0238 not found: ID does not exist"
Oct 30 23:15:40.492144 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.492105    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"]
Oct 30 23:15:40.507632 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:40.507599    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-7d79587f48-cng2s"]
Oct 30 23:15:41.290217 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:41.290173    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" path="/var/lib/kubelet/pods/c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40/volumes"
Oct 30 23:15:46.076974 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:46.076922    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:15:49.402141 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.402098    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"]
Oct 30 23:15:49.402744 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.402724    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main"
Oct 30 23:15:49.402795 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.402748    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main"
Oct 30 23:15:49.402795 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.402773    5060 cpu_manager.go:414] "containerMap: removing container" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="storage-initializer"
Oct 30 23:15:49.402795 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.402781    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="storage-initializer"
Oct 30 23:15:49.402929 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.402916    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1479a96-d4ac-46cc-a8f9-ac1b1d31fe40" containerName="main"
Oct 30 23:15:49.406833 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.406808    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.411386 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.411355    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Oct 30 23:15:49.411549 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.411410    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Oct 30 23:15:49.411549 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.411449    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:15:49.419216 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.419186    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"]
Oct 30 23:15:49.440625 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.440588    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Oct 30 23:15:49.440625 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.440588    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-kmq9c\""
Oct 30 23:15:49.440832 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.440589    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Oct 30 23:15:49.515428 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.515389    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-model-cache\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.515428 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.515429    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d779ec13-27e9-4baf-8dcc-24dd90574b05-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.515663 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.515541    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.515663 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.515590    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-dshm\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.515663 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.515642    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4rt\" (UniqueName: \"kubernetes.io/projected/d779ec13-27e9-4baf-8dcc-24dd90574b05-kube-api-access-zc4rt\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.515757 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.515682    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.515757 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.515708    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-home\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.515757 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.515741    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d779ec13-27e9-4baf-8dcc-24dd90574b05-tls-certs\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.616576 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.616531    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-model-cache\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.616576 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.616581    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d779ec13-27e9-4baf-8dcc-24dd90574b05-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.616838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.616606    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.616838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.616624    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-dshm\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.616838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.616642    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4rt\" (UniqueName: \"kubernetes.io/projected/d779ec13-27e9-4baf-8dcc-24dd90574b05-kube-api-access-zc4rt\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.616838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.616665    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.616838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.616696    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-home\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.616838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.616720    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d779ec13-27e9-4baf-8dcc-24dd90574b05-tls-certs\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.617176 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.616947    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-model-cache\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.617176 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.616986    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.617283 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.617228    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.617338 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.617292    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-home\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.617443 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.617419    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d779ec13-27e9-4baf-8dcc-24dd90574b05-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.619138 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.619110    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-dshm\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.619254 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.619177    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d779ec13-27e9-4baf-8dcc-24dd90574b05-tls-certs\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.630424 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.630389    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4rt\" (UniqueName: \"kubernetes.io/projected/d779ec13-27e9-4baf-8dcc-24dd90574b05-kube-api-access-zc4rt\") pod \"lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:49.724429 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:49.724388    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:50.013586 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:50.013477    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"]
Oct 30 23:15:50.017248 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:15:50.017222    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd779ec13_27e9_4baf_8dcc_24dd90574b05.slice/crio-e6d15cdd23c87c1ba566dcff58d43bd6154d93359852b26ed0546e2e8797e9db WatchSource:0}: Error finding container e6d15cdd23c87c1ba566dcff58d43bd6154d93359852b26ed0546e2e8797e9db: Status 404 returned error can't find the container with id e6d15cdd23c87c1ba566dcff58d43bd6154d93359852b26ed0546e2e8797e9db
Oct 30 23:15:50.528194 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:50.528148    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" event={"ID":"d779ec13-27e9-4baf-8dcc-24dd90574b05","Type":"ContainerStarted","Data":"c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550"}
Oct 30 23:15:50.528910 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:50.528861    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" event={"ID":"d779ec13-27e9-4baf-8dcc-24dd90574b05","Type":"ContainerStarted","Data":"e6d15cdd23c87c1ba566dcff58d43bd6154d93359852b26ed0546e2e8797e9db"}
Oct 30 23:15:54.548598 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:54.548562    5060 generic.go:356] "Generic (PLEG): container finished" podID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerID="c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550" exitCode=0
Oct 30 23:15:54.548976 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:54.548643    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" event={"ID":"d779ec13-27e9-4baf-8dcc-24dd90574b05","Type":"ContainerDied","Data":"c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550"}
Oct 30 23:15:55.556166 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:55.556127    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" event={"ID":"d779ec13-27e9-4baf-8dcc-24dd90574b05","Type":"ContainerStarted","Data":"cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c"}
Oct 30 23:15:55.587591 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:55.587526    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" podStartSLOduration=3.045806918 podStartE2EDuration="6.587488854s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.541681936s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:15:55.583818438 +0000 UTC m=+3436.403556269" watchObservedRunningTime="2026-10-30 23:15:55.587488854 +0000 UTC m=+3436.407226674"
Oct 30 23:15:55.724923 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:55.724880    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:56.077394 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:56.077347    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:15:59.724827 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:59.724785    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:15:59.726667 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:15:59.726630    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 30 23:16:04.778789 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:16:04.778746    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:16:06.077199 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:06.077153    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:16:09.725891 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:09.725833    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 30 23:16:16.076855 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:16.076795    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:16:19.725281 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:19.725234    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 30 23:16:26.076714 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:26.076588    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:16:29.725118 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:29.725070    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 30 23:16:34.778769 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:16:34.778723    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:16:36.077013 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:36.076964    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Oct 30 23:16:39.725631 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:39.725589    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 30 23:16:41.024020 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:41.023965    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" podUID="a5c727ac326546386befce51c4d99c6d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 23:16:46.092139 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:46.092107    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:16:46.100011 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:46.099970    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:16:49.725734 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:49.725680    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 30 23:16:54.982279 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:54.982240    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"]
Oct 30 23:16:59.725741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:16:59.725695    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 30 23:17:04.777241 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:17:04.777196    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:17:09.725345 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:09.725303    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 30 23:17:10.005419 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.005318    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main" containerID="cri-o://9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3" gracePeriod=45
Oct 30 23:17:10.762633 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.762602    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:17:10.825449 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.825354    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kserve-provision-location\") pod \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") "
Oct 30 23:17:10.825449 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.825403    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tls-certs\") pod \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") "
Oct 30 23:17:10.825449 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.825424    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-dshm\") pod \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") "
Oct 30 23:17:10.825750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.825455    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kube-api-access-vr4n8\" (UniqueName: \"kubernetes.io/projected/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kube-api-access-vr4n8\") pod \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") "
Oct 30 23:17:10.825750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.825544    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-model-cache\") pod \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") "
Oct 30 23:17:10.825750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.825575    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-home\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-home\") pod \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") "
Oct 30 23:17:10.825750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.825618    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-cabundle-cert\") pod \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") "
Oct 30 23:17:10.825750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.825646    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tmp-dir\") pod \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\" (UID: \"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3\") "
Oct 30 23:17:10.826203 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.826137    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-model-cache" pod "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" (UID: "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:10.826286 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.826205    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-cabundle-cert" pod "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" (UID: "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:17:10.826591 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.826552    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-home" pod "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" (UID: "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:10.828372 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.828334    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-dshm" pod "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" (UID: "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:10.828474 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.828440    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kube-api-access-vr4n8" pod "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" (UID: "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3"). InnerVolumeSpecName "kube-api-access-vr4n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:17:10.828658 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.828630    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tls-certs" pod "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" (UID: "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:17:10.838375 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.838342    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tmp-dir" pod "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" (UID: "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:10.855000 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.854959    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kserve-provision-location" pod "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" (UID: "38ed9f19-6e5e-46f5-8d7a-c318c4c405a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:17:10.927481 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.927442    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:17:10.927481 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.927475    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:17:10.927481 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.927485    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:17:10.927731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.927494    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr4n8\" (UniqueName: \"kubernetes.io/projected/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-kube-api-access-vr4n8\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:17:10.927731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.927527    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:17:10.927731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.927540    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:17:10.927731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.927553    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:17:10.927731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.927565    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:17:10.964644 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.964595    5060 generic.go:356] "Generic (PLEG): container finished" podID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerID="9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3" exitCode=0
Oct 30 23:17:10.964819 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.964654    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"
Oct 30 23:17:10.964819 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.964656    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" event={"ID":"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3","Type":"ContainerDied","Data":"9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3"}
Oct 30 23:17:10.964819 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.964711    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f" event={"ID":"38ed9f19-6e5e-46f5-8d7a-c318c4c405a3","Type":"ContainerDied","Data":"20c402a6ad6af2d94e613a12d3e44feebd86d73f067e429b7e00fdef6ca20481"}
Oct 30 23:17:10.964819 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.964732    5060 scope.go:122] "RemoveContainer" containerID="9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3"
Oct 30 23:17:10.975288 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.975251    5060 scope.go:122] "RemoveContainer" containerID="a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81"
Oct 30 23:17:10.983490 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.983469    5060 scope.go:122] "RemoveContainer" containerID="9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3"
Oct 30 23:17:10.984343 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:17:10.984322    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3\": container with ID starting with 9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3 not found: ID does not exist" containerID="9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3"
Oct 30 23:17:10.984388 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.984352    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3"} err="failed to get container status \"9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3\": rpc error: code = NotFound desc = could not find container \"9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3\": container with ID starting with 9b24271532e206e03076fd9fcb79db960372630ab8b984edab63a99e81c4c5f3 not found: ID does not exist"
Oct 30 23:17:10.984388 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.984370    5060 scope.go:122] "RemoveContainer" containerID="a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81"
Oct 30 23:17:10.984630 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:17:10.984615    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81\": container with ID starting with a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81 not found: ID does not exist" containerID="a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81"
Oct 30 23:17:10.984670 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.984634    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81"} err="failed to get container status \"a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81\": rpc error: code = NotFound desc = could not find container \"a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81\": container with ID starting with a2233bc59cd9b1e1d5bd355bbd336374e048ec336706b7dfbe3f85161b5f6f81 not found: ID does not exist"
Oct 30 23:17:10.992063 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.992037    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"]
Oct 30 23:17:10.998450 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:10.998423    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6ct755f"]
Oct 30 23:17:11.289901 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:11.289866    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" path="/var/lib/kubelet/pods/38ed9f19-6e5e-46f5-8d7a-c318c4c405a3/volumes"
Oct 30 23:17:19.734966 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:19.734927    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:17:19.742862 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:19.742822    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:17:34.776900 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:17:34.776798    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:17:46.504706 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:46.504670    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5_d779ec13-27e9-4baf-8dcc-24dd90574b05/storage-initializer/0.log"
Oct 30 23:17:46.531663 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:46.531629    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5_d779ec13-27e9-4baf-8dcc-24dd90574b05/main/0.log"
Oct 30 23:17:47.469345 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.469313    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"]
Oct 30 23:17:47.469791 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.469772    5060 cpu_manager.go:414] "containerMap: removing container" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="storage-initializer"
Oct 30 23:17:47.469894 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.469793    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="storage-initializer"
Oct 30 23:17:47.469894 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.469823    5060 cpu_manager.go:414] "containerMap: removing container" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main"
Oct 30 23:17:47.469894 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.469831    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main"
Oct 30 23:17:47.470051 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.469978    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ed9f19-6e5e-46f5-8d7a-c318c4c405a3" containerName="main"
Oct 30 23:17:47.473818 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.473795    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.477418 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:17:47.477377    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" is forbidden: User \"system:node:ip-10-0-138-251.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-d73d44f4\": no relationship found between node 'ip-10-0-138-251.us-east-2.compute.internal' and this object" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.479884 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.479865    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-qbdtm\""
Oct 30 23:17:47.480091 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.480077    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Oct 30 23:17:47.480377 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.480346    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:17:47.480949 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.480929    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Oct 30 23:17:47.482669 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.482648    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Oct 30 23:17:47.497394 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.497363    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"]
Oct 30 23:17:47.570148 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.570110    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Oct 30 23:17:47.595603 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.595560    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.595750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.595609    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.595750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.595672    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.595750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.595713    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.595750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.595738    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.595880 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.595830    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.595880 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.595860    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.595942 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.595881    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8jn\" (UniqueName: \"kubernetes.io/projected/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kube-api-access-cb8jn\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.696561 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.696488    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.696744 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.696573    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.696744 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.696599    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.696744 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.696648    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.696744 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.696685    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.696744 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.696738    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.696960 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.696769    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.696960 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.696932    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8jn\" (UniqueName: \"kubernetes.io/projected/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kube-api-access-cb8jn\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.697061 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.697031    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.697175 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.697090    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.697175 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.697124    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.697309 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.697288    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.697493 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.697473    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.699072 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.699054    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.699252 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.699235    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.714600 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.714555    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8jn\" (UniqueName: \"kubernetes.io/projected/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kube-api-access-cb8jn\") pod \"custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:47.792653 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:47.792562    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:48.087175 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:48.087144    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"]
Oct 30 23:17:48.090560 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:17:48.090527    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6feb6a_2a21_4f0e_8837_eaf604dab7fc.slice/crio-af5defba4c743ca471b339fa133de8784ddb95ee6dc78755810b5df436886607 WatchSource:0}: Error finding container af5defba4c743ca471b339fa133de8784ddb95ee6dc78755810b5df436886607: Status 404 returned error can't find the container with id af5defba4c743ca471b339fa133de8784ddb95ee6dc78755810b5df436886607
Oct 30 23:17:48.145565 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:48.145369    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" event={"ID":"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc","Type":"ContainerStarted","Data":"af5defba4c743ca471b339fa133de8784ddb95ee6dc78755810b5df436886607"}
Oct 30 23:17:49.153106 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:49.153068    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" event={"ID":"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc","Type":"ContainerStarted","Data":"ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992"}
Oct 30 23:17:50.734182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:50.734141    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"]
Oct 30 23:17:51.165173 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:51.165080    5060 generic.go:356] "Generic (PLEG): container finished" podID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerID="ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992" exitCode=0
Oct 30 23:17:51.165173 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:51.165151    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" event={"ID":"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc","Type":"ContainerDied","Data":"ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992"}
Oct 30 23:17:52.172593 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:52.172551    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" event={"ID":"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc","Type":"ContainerStarted","Data":"6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271"}
Oct 30 23:17:52.205557 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:52.205474    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podStartSLOduration=2.630486224 podStartE2EDuration="5.205461221s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.574974997s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:17:52.205312821 +0000 UTC m=+3553.025050651" watchObservedRunningTime="2026-10-30 23:17:52.205461221 +0000 UTC m=+3553.025199042"
Oct 30 23:17:57.793367 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:57.793311    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:57.793834 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:57.793405    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:17:57.795019 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:17:57.794986    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:18:04.776169 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:18:04.776130    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:18:05.760865 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:05.760794    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main" containerID="cri-o://cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c" gracePeriod=45
Oct 30 23:18:06.433780 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.433751    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:18:06.482745 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.482709    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-model-cache\") pod \"d779ec13-27e9-4baf-8dcc-24dd90574b05\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") "
Oct 30 23:18:06.482937 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.482755    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-home\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-home\") pod \"d779ec13-27e9-4baf-8dcc-24dd90574b05\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") "
Oct 30 23:18:06.482937 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.482777    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-kserve-provision-location\") pod \"d779ec13-27e9-4baf-8dcc-24dd90574b05\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") "
Oct 30 23:18:06.482937 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.482804    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-dshm\") pod \"d779ec13-27e9-4baf-8dcc-24dd90574b05\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") "
Oct 30 23:18:06.482937 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.482855    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d779ec13-27e9-4baf-8dcc-24dd90574b05-tls-certs\" (UniqueName: \"kubernetes.io/secret/d779ec13-27e9-4baf-8dcc-24dd90574b05-tls-certs\") pod \"d779ec13-27e9-4baf-8dcc-24dd90574b05\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") "
Oct 30 23:18:06.482937 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.482902    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-tmp-dir\") pod \"d779ec13-27e9-4baf-8dcc-24dd90574b05\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") "
Oct 30 23:18:06.483242 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.482997    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d779ec13-27e9-4baf-8dcc-24dd90574b05-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d779ec13-27e9-4baf-8dcc-24dd90574b05-cabundle-cert\") pod \"d779ec13-27e9-4baf-8dcc-24dd90574b05\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") "
Oct 30 23:18:06.483242 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.483029    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d779ec13-27e9-4baf-8dcc-24dd90574b05-kube-api-access-zc4rt\" (UniqueName: \"kubernetes.io/projected/d779ec13-27e9-4baf-8dcc-24dd90574b05-kube-api-access-zc4rt\") pod \"d779ec13-27e9-4baf-8dcc-24dd90574b05\" (UID: \"d779ec13-27e9-4baf-8dcc-24dd90574b05\") "
Oct 30 23:18:06.483242 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.483076    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-model-cache" pod "d779ec13-27e9-4baf-8dcc-24dd90574b05" (UID: "d779ec13-27e9-4baf-8dcc-24dd90574b05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:06.483242 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.483211    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-home" pod "d779ec13-27e9-4baf-8dcc-24dd90574b05" (UID: "d779ec13-27e9-4baf-8dcc-24dd90574b05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:06.483463 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.483373    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:18:06.483719 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.483651    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d779ec13-27e9-4baf-8dcc-24dd90574b05-cabundle-cert" pod "d779ec13-27e9-4baf-8dcc-24dd90574b05" (UID: "d779ec13-27e9-4baf-8dcc-24dd90574b05"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:18:06.485421 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.485393    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d779ec13-27e9-4baf-8dcc-24dd90574b05-tls-certs" pod "d779ec13-27e9-4baf-8dcc-24dd90574b05" (UID: "d779ec13-27e9-4baf-8dcc-24dd90574b05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:18:06.485694 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.485672    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-dshm" pod "d779ec13-27e9-4baf-8dcc-24dd90574b05" (UID: "d779ec13-27e9-4baf-8dcc-24dd90574b05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:06.485780 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.485749    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d779ec13-27e9-4baf-8dcc-24dd90574b05-kube-api-access-zc4rt" pod "d779ec13-27e9-4baf-8dcc-24dd90574b05" (UID: "d779ec13-27e9-4baf-8dcc-24dd90574b05"). InnerVolumeSpecName "kube-api-access-zc4rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:18:06.503454 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.503410    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-tmp-dir" pod "d779ec13-27e9-4baf-8dcc-24dd90574b05" (UID: "d779ec13-27e9-4baf-8dcc-24dd90574b05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:06.520645 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.520598    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-kserve-provision-location" pod "d779ec13-27e9-4baf-8dcc-24dd90574b05" (UID: "d779ec13-27e9-4baf-8dcc-24dd90574b05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:18:06.584687 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.584645    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:18:06.584687 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.584681    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d779ec13-27e9-4baf-8dcc-24dd90574b05-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:18:06.584898 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.584698    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:18:06.584898 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.584711    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d779ec13-27e9-4baf-8dcc-24dd90574b05-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:18:06.584898 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.584725    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc4rt\" (UniqueName: \"kubernetes.io/projected/d779ec13-27e9-4baf-8dcc-24dd90574b05-kube-api-access-zc4rt\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:18:06.584898 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.584738    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:18:06.584898 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:06.584749    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d779ec13-27e9-4baf-8dcc-24dd90574b05-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:18:07.254596 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.254556    5060 generic.go:356] "Generic (PLEG): container finished" podID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerID="cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c" exitCode=0
Oct 30 23:18:07.254813 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.254621    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"
Oct 30 23:18:07.254813 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.254642    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" event={"ID":"d779ec13-27e9-4baf-8dcc-24dd90574b05","Type":"ContainerDied","Data":"cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c"}
Oct 30 23:18:07.254813 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.254719    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5" event={"ID":"d779ec13-27e9-4baf-8dcc-24dd90574b05","Type":"ContainerDied","Data":"e6d15cdd23c87c1ba566dcff58d43bd6154d93359852b26ed0546e2e8797e9db"}
Oct 30 23:18:07.254813 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.254741    5060 scope.go:122] "RemoveContainer" containerID="cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c"
Oct 30 23:18:07.265650 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.265622    5060 scope.go:122] "RemoveContainer" containerID="c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550"
Oct 30 23:18:07.274397 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.274375    5060 scope.go:122] "RemoveContainer" containerID="cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c"
Oct 30 23:18:07.274779 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:18:07.274757    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c\": container with ID starting with cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c not found: ID does not exist" containerID="cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c"
Oct 30 23:18:07.274835 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.274793    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c"} err="failed to get container status \"cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c\": rpc error: code = NotFound desc = could not find container \"cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c\": container with ID starting with cbb4de82a4aead3b8401ddf535b73b4847cc1358ce317d210060285b7594920c not found: ID does not exist"
Oct 30 23:18:07.274835 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.274811    5060 scope.go:122] "RemoveContainer" containerID="c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550"
Oct 30 23:18:07.275072 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:18:07.275055    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550\": container with ID starting with c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550 not found: ID does not exist" containerID="c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550"
Oct 30 23:18:07.275130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.275084    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550"} err="failed to get container status \"c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550\": rpc error: code = NotFound desc = could not find container \"c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550\": container with ID starting with c6df98a7eda57abd0318b84ec65a7ef6b34af35b148b7f1b1ed5e6c8bb46c550 not found: ID does not exist"
Oct 30 23:18:07.281370 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.281340    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"]
Oct 30 23:18:07.289993 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.289965    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-769c5d65fc-p2mf5"]
Oct 30 23:18:07.793576 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:07.793532    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:18:09.290419 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:09.290372    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" path="/var/lib/kubelet/pods/d779ec13-27e9-4baf-8dcc-24dd90574b05/volumes"
Oct 30 23:18:17.793290 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:17.793235    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:18:27.793706 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:27.793660    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:18:34.777166 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:18:34.777122    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:18:37.793125 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:37.793073    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:18:39.722220 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:39.722187    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:18:39.766550 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:39.766489    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:18:41.023893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:41.023836    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" podUID="a5c727ac326546386befce51c4d99c6d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 23:18:47.793009 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:47.792954    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:18:54.926519 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:54.926466    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"]
Oct 30 23:18:54.927146 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:54.926722    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerName="istio-proxy" containerID="cri-o://036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9" gracePeriod=30
Oct 30 23:18:54.928166 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:54.928135    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"]
Oct 30 23:18:54.928389 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:54.928364    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerName="istio-proxy" containerID="cri-o://7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8" gracePeriod=30
Oct 30 23:18:54.929644 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:54.929619    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"]
Oct 30 23:18:54.929895 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:54.929837    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" podUID="44629e36-a027-4fac-9839-b268be7f21d7" containerName="istio-proxy" containerID="cri-o://585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35" gracePeriod=30
Oct 30 23:18:57.793173 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:57.793121    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:18:58.849586 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:58.849541    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-ghl2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:18:58.850111 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:18:58.849622    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" podUID="44629e36-a027-4fac-9839-b268be7f21d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:19:00.200179 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.200084    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:19:00.238651 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.238622    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:19:00.246337 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.246307    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 23:19:00.289362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289110    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-envoy\") pod \"44629e36-a027-4fac-9839-b268be7f21d7\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") "
Oct 30 23:19:00.289362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289163    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-kube-api-access-bqk2q\" (UniqueName: \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-kube-api-access-bqk2q\") pod \"5fb5a636-f3dd-4961-bc4f-da402e02b620\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") "
Oct 30 23:19:00.289362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289201    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-podinfo\") pod \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") "
Oct 30 23:19:00.289362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289240    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-credential-socket\") pod \"44629e36-a027-4fac-9839-b268be7f21d7\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") "
Oct 30 23:19:00.289362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289268    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-token\" (UniqueName: \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-token\") pod \"5fb5a636-f3dd-4961-bc4f-da402e02b620\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") "
Oct 30 23:19:00.289362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289299    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-certs\") pod \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") "
Oct 30 23:19:00.289362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289327    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44629e36-a027-4fac-9839-b268be7f21d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44629e36-a027-4fac-9839-b268be7f21d7-istiod-ca-cert\") pod \"44629e36-a027-4fac-9839-b268be7f21d7\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") "
Oct 30 23:19:00.289362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289362    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-socket\") pod \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289392    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44629e36-a027-4fac-9839-b268be7f21d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44629e36-a027-4fac-9839-b268be7f21d7-istio-podinfo\") pod \"44629e36-a027-4fac-9839-b268be7f21d7\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289421    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-data\") pod \"5fb5a636-f3dd-4961-bc4f-da402e02b620\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289467    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-kube-api-access-jx7wt\" (UniqueName: \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-kube-api-access-jx7wt\") pod \"44629e36-a027-4fac-9839-b268be7f21d7\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289538    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-kube-api-access-5dd4b\" (UniqueName: \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-kube-api-access-5dd4b\") pod \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289567    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-credential-socket\") pod \"5fb5a636-f3dd-4961-bc4f-da402e02b620\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289600    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-certs\") pod \"44629e36-a027-4fac-9839-b268be7f21d7\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289628    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-istio-token\" (UniqueName: \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-istio-token\") pod \"44629e36-a027-4fac-9839-b268be7f21d7\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289654    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-envoy\") pod \"5fb5a636-f3dd-4961-bc4f-da402e02b620\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289687    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-envoy\") pod \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289717    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istiod-ca-cert\") pod \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289763    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-socket\") pod \"44629e36-a027-4fac-9839-b268be7f21d7\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289811    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-token\" (UniqueName: \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-token\") pod \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289843    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-data\") pod \"44629e36-a027-4fac-9839-b268be7f21d7\" (UID: \"44629e36-a027-4fac-9839-b268be7f21d7\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289870    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-credential-socket\") pod \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289898    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-data\") pod \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\" (UID: \"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289925    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-certs\") pod \"5fb5a636-f3dd-4961-bc4f-da402e02b620\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.289963    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-socket\") pod \"5fb5a636-f3dd-4961-bc4f-da402e02b620\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.290017    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb5a636-f3dd-4961-bc4f-da402e02b620-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb5a636-f3dd-4961-bc4f-da402e02b620-istiod-ca-cert\") pod \"5fb5a636-f3dd-4961-bc4f-da402e02b620\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.290049    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-podinfo\") pod \"5fb5a636-f3dd-4961-bc4f-da402e02b620\" (UID: \"5fb5a636-f3dd-4961-bc4f-da402e02b620\") "
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.290486    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-credential-socket" pod "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" (UID: "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.290755    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-socket" pod "44629e36-a027-4fac-9839-b268be7f21d7" (UID: "44629e36-a027-4fac-9839-b268be7f21d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.290938    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-credential-socket" pod "44629e36-a027-4fac-9839-b268be7f21d7" (UID: "44629e36-a027-4fac-9839-b268be7f21d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.291172    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istiod-ca-cert" pod "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" (UID: "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.291348    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-certs" pod "44629e36-a027-4fac-9839-b268be7f21d7" (UID: "44629e36-a027-4fac-9839-b268be7f21d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.291374    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-credential-socket" pod "5fb5a636-f3dd-4961-bc4f-da402e02b620" (UID: "5fb5a636-f3dd-4961-bc4f-da402e02b620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.292170    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-data" pod "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" (UID: "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.292352    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-certs" pod "5fb5a636-f3dd-4961-bc4f-da402e02b620" (UID: "5fb5a636-f3dd-4961-bc4f-da402e02b620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.292548    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-socket" pod "5fb5a636-f3dd-4961-bc4f-da402e02b620" (UID: "5fb5a636-f3dd-4961-bc4f-da402e02b620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.292835    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb5a636-f3dd-4961-bc4f-da402e02b620-istiod-ca-cert" pod "5fb5a636-f3dd-4961-bc4f-da402e02b620" (UID: "5fb5a636-f3dd-4961-bc4f-da402e02b620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.292883    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-data" pod "5fb5a636-f3dd-4961-bc4f-da402e02b620" (UID: "5fb5a636-f3dd-4961-bc4f-da402e02b620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.292967    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-podinfo" pod "5fb5a636-f3dd-4961-bc4f-da402e02b620" (UID: "5fb5a636-f3dd-4961-bc4f-da402e02b620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.293204    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-data" pod "44629e36-a027-4fac-9839-b268be7f21d7" (UID: "44629e36-a027-4fac-9839-b268be7f21d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.293709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.293405    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-certs" pod "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" (UID: "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.296963 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.295817    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44629e36-a027-4fac-9839-b268be7f21d7-istiod-ca-cert" pod "44629e36-a027-4fac-9839-b268be7f21d7" (UID: "44629e36-a027-4fac-9839-b268be7f21d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:19:00.296963 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.296147    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-socket" pod "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" (UID: "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.296963 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.296358    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-kube-api-access-bqk2q" pod "5fb5a636-f3dd-4961-bc4f-da402e02b620" (UID: "5fb5a636-f3dd-4961-bc4f-da402e02b620"). InnerVolumeSpecName "kube-api-access-bqk2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:00.297447 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.297406    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-token" pod "5fb5a636-f3dd-4961-bc4f-da402e02b620" (UID: "5fb5a636-f3dd-4961-bc4f-da402e02b620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:00.297447 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.297419    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-istio-token" pod "44629e36-a027-4fac-9839-b268be7f21d7" (UID: "44629e36-a027-4fac-9839-b268be7f21d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:00.297780 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.297749    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-envoy" pod "44629e36-a027-4fac-9839-b268be7f21d7" (UID: "44629e36-a027-4fac-9839-b268be7f21d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.297879 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.297843    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-token" pod "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" (UID: "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:00.298499 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.298471    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-podinfo" pod "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" (UID: "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:19:00.299097 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.299066    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-envoy" pod "5fb5a636-f3dd-4961-bc4f-da402e02b620" (UID: "5fb5a636-f3dd-4961-bc4f-da402e02b620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.299182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.299138    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-kube-api-access-5dd4b" pod "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" (UID: "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3"). InnerVolumeSpecName "kube-api-access-5dd4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:00.299246 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.299191    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44629e36-a027-4fac-9839-b268be7f21d7-istio-podinfo" pod "44629e36-a027-4fac-9839-b268be7f21d7" (UID: "44629e36-a027-4fac-9839-b268be7f21d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:19:00.299595 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.299573    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-envoy" pod "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" (UID: "e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:19:00.299676 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.299609    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-kube-api-access-jx7wt" pod "44629e36-a027-4fac-9839-b268be7f21d7" (UID: "44629e36-a027-4fac-9839-b268be7f21d7"). InnerVolumeSpecName "kube-api-access-jx7wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:19:00.391069 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391029    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391069 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391071    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391086    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391102    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391116    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391129    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391143    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb5a636-f3dd-4961-bc4f-da402e02b620-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391156    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391169    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391183    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqk2q\" (UniqueName: \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-kube-api-access-bqk2q\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391199    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391213    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391233    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391246    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391261    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44629e36-a027-4fac-9839-b268be7f21d7-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391274    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391288    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44629e36-a027-4fac-9839-b268be7f21d7-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391302    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391315    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx7wt\" (UniqueName: \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-kube-api-access-jx7wt\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391328    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dd4b\" (UniqueName: \"kubernetes.io/projected/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-kube-api-access-5dd4b\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391341    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391353    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391365    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44629e36-a027-4fac-9839-b268be7f21d7-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391378    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb5a636-f3dd-4961-bc4f-da402e02b620-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391393    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391406    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.391809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.391421    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44629e36-a027-4fac-9839-b268be7f21d7-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:19:00.513543 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.513428    5060 generic.go:356] "Generic (PLEG): container finished" podID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerID="036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9" exitCode=0
Oct 30 23:19:00.513543 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.513494    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" event={"ID":"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3","Type":"ContainerDied","Data":"036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9"}
Oct 30 23:19:00.513775 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.513561    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j" event={"ID":"e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3","Type":"ContainerDied","Data":"f91edec0c694d105c172f9398e7f7fe4f43f658b45976575c3898c94a21a5314"}
Oct 30 23:19:00.513775 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.513571    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"
Oct 30 23:19:00.513775 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.513582    5060 scope.go:122] "RemoveContainer" containerID="036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9"
Oct 30 23:19:00.515207 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.515186    5060 generic.go:356] "Generic (PLEG): container finished" podID="44629e36-a027-4fac-9839-b268be7f21d7" containerID="585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35" exitCode=0
Oct 30 23:19:00.515347 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.515245    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"
Oct 30 23:19:00.515347 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.515269    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" event={"ID":"44629e36-a027-4fac-9839-b268be7f21d7","Type":"ContainerDied","Data":"585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35"}
Oct 30 23:19:00.515347 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.515339    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l" event={"ID":"44629e36-a027-4fac-9839-b268be7f21d7","Type":"ContainerDied","Data":"b66499111c3fc05c5e0207b4cb28dfa71b799cf9daabc4110d808732c57bb68e"}
Oct 30 23:19:00.516680 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.516658    5060 generic.go:356] "Generic (PLEG): container finished" podID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerID="7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8" exitCode=0
Oct 30 23:19:00.516786 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.516748    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" event={"ID":"5fb5a636-f3dd-4961-bc4f-da402e02b620","Type":"ContainerDied","Data":"7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8"}
Oct 30 23:19:00.516786 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.516778    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj" event={"ID":"5fb5a636-f3dd-4961-bc4f-da402e02b620","Type":"ContainerDied","Data":"38dd2340287c49a814a6b2dd40cbfc30ecaf3235dc76a27ffd7a9f1ba88165d3"}
Oct 30 23:19:00.516899 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.516827    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"
Oct 30 23:19:00.524431 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.524409    5060 scope.go:122] "RemoveContainer" containerID="036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9"
Oct 30 23:19:00.524912 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:19:00.524762    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9\": container with ID starting with 036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9 not found: ID does not exist" containerID="036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9"
Oct 30 23:19:00.524912 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.524801    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9"} err="failed to get container status \"036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9\": rpc error: code = NotFound desc = could not find container \"036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9\": container with ID starting with 036edc92fc487e58a80631ea1fa318b66b670b45be4e28af1de6425035599ab9 not found: ID does not exist"
Oct 30 23:19:00.524912 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.524828    5060 scope.go:122] "RemoveContainer" containerID="585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35"
Oct 30 23:19:00.534870 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.534850    5060 scope.go:122] "RemoveContainer" containerID="585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35"
Oct 30 23:19:00.535187 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:19:00.535166    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35\": container with ID starting with 585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35 not found: ID does not exist" containerID="585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35"
Oct 30 23:19:00.535246 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.535195    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35"} err="failed to get container status \"585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35\": rpc error: code = NotFound desc = could not find container \"585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35\": container with ID starting with 585665ee69126b5e6484ab605d2ace14e38d951395230c2227ba332c7c662f35 not found: ID does not exist"
Oct 30 23:19:00.535246 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.535212    5060 scope.go:122] "RemoveContainer" containerID="7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8"
Oct 30 23:19:00.544589 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.544541    5060 scope.go:122] "RemoveContainer" containerID="7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8"
Oct 30 23:19:00.545127 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:19:00.544948    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8\": container with ID starting with 7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8 not found: ID does not exist" containerID="7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8"
Oct 30 23:19:00.545127 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.544985    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8"} err="failed to get container status \"7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8\": rpc error: code = NotFound desc = could not find container \"7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8\": container with ID starting with 7578a3a960c74599331c4959432253a12614b10a831e2dc28c5d887b3a76fdf8 not found: ID does not exist"
Oct 30 23:19:00.549036 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.549010    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"]
Oct 30 23:19:00.550456 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.550437    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-x9w7j"]
Oct 30 23:19:00.563207 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.563178    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"]
Oct 30 23:19:00.570352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.570320    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghl2l"]
Oct 30 23:19:00.584281 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.584245    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"]
Oct 30 23:19:00.589385 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:00.589355    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-ghfbj"]
Oct 30 23:19:01.290536 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:01.290440    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44629e36-a027-4fac-9839-b268be7f21d7" path="/var/lib/kubelet/pods/44629e36-a027-4fac-9839-b268be7f21d7/volumes"
Oct 30 23:19:01.290906 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:01.290888    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" path="/var/lib/kubelet/pods/5fb5a636-f3dd-4961-bc4f-da402e02b620/volumes"
Oct 30 23:19:01.291279 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:01.291265    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" path="/var/lib/kubelet/pods/e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3/volumes"
Oct 30 23:19:04.776351 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:19:04.776301    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:19:07.793348 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:07.793296    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:19:17.793469 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:17.793424    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:19:27.793488 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:27.793398    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:19:34.778057 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:19:34.778006    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:19:37.793097 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:37.793041    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 30 23:19:47.802731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:47.802690    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:19:47.810473 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:47.810443    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:19:56.516711 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:19:56.516674    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"]
Oct 30 23:20:04.775714 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:20:04.775674    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:20:11.539463 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:11.539416    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main" containerID="cri-o://6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271" gracePeriod=45
Oct 30 23:20:12.285670 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.285642    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:20:12.441516 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.441479    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-dshm\") pod \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") "
Oct 30 23:20:12.441716 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.441539    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-model-cache\") pod \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") "
Oct 30 23:20:12.441716 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.441581    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kserve-provision-location\") pod \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") "
Oct 30 23:20:12.441716 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.441614    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kube-api-access-cb8jn\" (UniqueName: \"kubernetes.io/projected/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kube-api-access-cb8jn\") pod \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") "
Oct 30 23:20:12.441716 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.441649    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-cabundle-cert\") pod \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") "
Oct 30 23:20:12.441716 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.441667    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-home\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-home\") pod \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") "
Oct 30 23:20:12.441716 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.441695    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tls-certs\") pod \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") "
Oct 30 23:20:12.442030 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.441898    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-model-cache" pod "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" (UID: "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:20:12.442081 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.442065    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tmp-dir\") pod \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\" (UID: \"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc\") "
Oct 30 23:20:12.442871 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.442377    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:12.442871 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.442554    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-cabundle-cert" pod "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" (UID: "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:20:12.443331 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.443304    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-home" pod "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" (UID: "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:20:12.444203 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.444170    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-dshm" pod "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" (UID: "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:20:12.444313 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.444231    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tls-certs" pod "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" (UID: "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:20:12.444375 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.444342    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kube-api-access-cb8jn" pod "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" (UID: "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc"). InnerVolumeSpecName "kube-api-access-cb8jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:20:12.456793 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.456757    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tmp-dir" pod "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" (UID: "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:20:12.466363 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.466326    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kserve-provision-location" pod "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" (UID: "3f6feb6a-2a21-4f0e-8837-eaf604dab7fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:20:12.543682 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.543643    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:12.543682 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.543674    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:12.543682 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.543686    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:12.544122 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.543697    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:12.544122 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.543708    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:12.544122 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.543721    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:12.544122 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.543735    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8jn\" (UniqueName: \"kubernetes.io/projected/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc-kube-api-access-cb8jn\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:12.855692 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.855609    5060 generic.go:356] "Generic (PLEG): container finished" podID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerID="6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271" exitCode=0
Oct 30 23:20:12.855692 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.855663    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" event={"ID":"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc","Type":"ContainerDied","Data":"6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271"}
Oct 30 23:20:12.855872 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.855714    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg" event={"ID":"3f6feb6a-2a21-4f0e-8837-eaf604dab7fc","Type":"ContainerDied","Data":"af5defba4c743ca471b339fa133de8784ddb95ee6dc78755810b5df436886607"}
Oct 30 23:20:12.855872 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.855719    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"
Oct 30 23:20:12.855872 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.855737    5060 scope.go:122] "RemoveContainer" containerID="6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271"
Oct 30 23:20:12.865293 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.865273    5060 scope.go:122] "RemoveContainer" containerID="ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992"
Oct 30 23:20:12.873264 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.873236    5060 scope.go:122] "RemoveContainer" containerID="6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271"
Oct 30 23:20:12.873562 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:20:12.873538    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271\": container with ID starting with 6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271 not found: ID does not exist" containerID="6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271"
Oct 30 23:20:12.873620 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.873574    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271"} err="failed to get container status \"6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271\": rpc error: code = NotFound desc = could not find container \"6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271\": container with ID starting with 6282999ee39670f1d41c6f66a2203f6dce20dc432c0d3ea4062044ad3f179271 not found: ID does not exist"
Oct 30 23:20:12.873620 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.873596    5060 scope.go:122] "RemoveContainer" containerID="ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992"
Oct 30 23:20:12.873847 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:20:12.873827    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992\": container with ID starting with ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992 not found: ID does not exist" containerID="ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992"
Oct 30 23:20:12.873900 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.873857    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992"} err="failed to get container status \"ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992\": rpc error: code = NotFound desc = could not find container \"ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992\": container with ID starting with ff2f412a9790ab190f4ddabbd66362831892949ce6b9a0b7ca63572a33b66992 not found: ID does not exist"
Oct 30 23:20:12.881441 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.881409    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"]
Oct 30 23:20:12.886481 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:12.886457    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-5f9bb68559-jp7pg"]
Oct 30 23:20:13.290563 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:13.290500    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" path="/var/lib/kubelet/pods/3f6feb6a-2a21-4f0e-8837-eaf604dab7fc/volumes"
Oct 30 23:20:23.584970 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.584926    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"]
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585274    5060 cpu_manager.go:414] "containerMap: removing container" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerName="istio-proxy"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585287    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerName="istio-proxy"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585297    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585304    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585316    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="storage-initializer"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585321    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="storage-initializer"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585329    5060 cpu_manager.go:414] "containerMap: removing container" podUID="44629e36-a027-4fac-9839-b268be7f21d7" containerName="istio-proxy"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585333    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44629e36-a027-4fac-9839-b268be7f21d7" containerName="istio-proxy"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585339    5060 cpu_manager.go:414] "containerMap: removing container" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585344    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585354    5060 cpu_manager.go:414] "containerMap: removing container" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerName="istio-proxy"
Oct 30 23:20:23.585352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585358    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerName="istio-proxy"
Oct 30 23:20:23.585745 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585378    5060 cpu_manager.go:414] "containerMap: removing container" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="storage-initializer"
Oct 30 23:20:23.585745 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585383    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="storage-initializer"
Oct 30 23:20:23.585745 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585435    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a6f7fa-efe5-4df1-a773-70cf8ca3d0c3" containerName="istio-proxy"
Oct 30 23:20:23.585745 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585443    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="d779ec13-27e9-4baf-8dcc-24dd90574b05" containerName="main"
Oct 30 23:20:23.585745 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585450    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="44629e36-a027-4fac-9839-b268be7f21d7" containerName="istio-proxy"
Oct 30 23:20:23.585745 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585456    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6feb6a-2a21-4f0e-8837-eaf604dab7fc" containerName="main"
Oct 30 23:20:23.585745 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.585461    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb5a636-f3dd-4961-bc4f-da402e02b620" containerName="istio-proxy"
Oct 30 23:20:23.588467 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.588450    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:23.593148 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.593122    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-mtvb8\""
Oct 30 23:20:23.595475 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.595437    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 30 23:20:23.596379 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.596351    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Oct 30 23:20:23.597650 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.597624    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Oct 30 23:20:23.597796 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.597774    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"]
Oct 30 23:20:23.736362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.736316    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zjx\" (UniqueName: \"kubernetes.io/projected/ddf0f043-7884-4364-b12a-241f33271ad3-kube-api-access-x8zjx\") pod \"e2e-pvc-lora-download-xvvcf\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:23.837725 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.837647    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zjx\" (UniqueName: \"kubernetes.io/projected/ddf0f043-7884-4364-b12a-241f33271ad3-kube-api-access-x8zjx\") pod \"e2e-pvc-lora-download-xvvcf\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:23.851467 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:23.851438    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zjx\" (UniqueName: \"kubernetes.io/projected/ddf0f043-7884-4364-b12a-241f33271ad3-kube-api-access-x8zjx\") pod \"e2e-pvc-lora-download-xvvcf\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:27.368172 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:27.368125    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"e2e-pvc-lora-download-xvvcf\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:27.375464 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:27.375434    5060 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"e2e-pvc-lora-download-xvvcf\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:27.468808 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:27.468771    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"e2e-pvc-lora-download-xvvcf\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:27.468966 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:27.468916    5060 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"e2e-pvc-lora-download-xvvcf\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:27.472939 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:27.472919    5060 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"e2e-pvc-lora-download-xvvcf\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") DevicePath \"csi-2938a377e19fedd411ba0532e0de3055c0d20a345c444844e77857b488b6d102\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:27.689560 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:27.689501    5060 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"e2e-pvc-lora-download-xvvcf\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a5028c51e1c80b1a87f02ed2b08bcca484e56aa9640f5321f7da4c8d94842e15/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:27.703610 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:27.703577    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"e2e-pvc-lora-download-xvvcf\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:27.800668 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:27.800603    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:28.091372 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:28.091342    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"]
Oct 30 23:20:28.093629 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:20:28.093599    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddf0f043_7884_4364_b12a_241f33271ad3.slice/crio-d0bec6f7cc97d29feaa818d7a8e58dd8d00c7f67b39d0dd28c26c2da123e568e WatchSource:0}: Error finding container d0bec6f7cc97d29feaa818d7a8e58dd8d00c7f67b39d0dd28c26c2da123e568e: Status 404 returned error can't find the container with id d0bec6f7cc97d29feaa818d7a8e58dd8d00c7f67b39d0dd28c26c2da123e568e
Oct 30 23:20:28.926756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:28.926718    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf" event={"ID":"ddf0f043-7884-4364-b12a-241f33271ad3","Type":"ContainerStarted","Data":"4490a433b1f80a57b4eac392aaa834dfe4c4a450689a88c349a20edd8b6f5c76"}
Oct 30 23:20:28.927155 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:28.926764    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf" event={"ID":"ddf0f043-7884-4364-b12a-241f33271ad3","Type":"ContainerStarted","Data":"d0bec6f7cc97d29feaa818d7a8e58dd8d00c7f67b39d0dd28c26c2da123e568e"}
Oct 30 23:20:28.945439 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:28.945372    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf" podStartSLOduration=8.945352873000001 podStartE2EDuration="8.945352873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:28.94456271 +0000 UTC m=+3709.764300532" watchObservedRunningTime="2026-10-30 23:20:28.945352873 +0000 UTC m=+3709.765090692"
Oct 30 23:20:30.936814 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:30.936779    5060 generic.go:356] "Generic (PLEG): container finished" podID="ddf0f043-7884-4364-b12a-241f33271ad3" containerID="4490a433b1f80a57b4eac392aaa834dfe4c4a450689a88c349a20edd8b6f5c76" exitCode=0
Oct 30 23:20:30.937201 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:30.936847    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf" event={"ID":"ddf0f043-7884-4364-b12a-241f33271ad3","Type":"ContainerDied","Data":"4490a433b1f80a57b4eac392aaa834dfe4c4a450689a88c349a20edd8b6f5c76"}
Oct 30 23:20:32.077839 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:32.077805    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:32.208571 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:32.208457    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"ddf0f043-7884-4364-b12a-241f33271ad3\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") "
Oct 30 23:20:32.208729 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:32.208578    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf0f043-7884-4364-b12a-241f33271ad3-kube-api-access-x8zjx\" (UniqueName: \"kubernetes.io/projected/ddf0f043-7884-4364-b12a-241f33271ad3-kube-api-access-x8zjx\") pod \"ddf0f043-7884-4364-b12a-241f33271ad3\" (UID: \"ddf0f043-7884-4364-b12a-241f33271ad3\") "
Oct 30 23:20:32.210810 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:32.210779    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf0f043-7884-4364-b12a-241f33271ad3-kube-api-access-x8zjx" pod "ddf0f043-7884-4364-b12a-241f33271ad3" (UID: "ddf0f043-7884-4364-b12a-241f33271ad3"). InnerVolumeSpecName "kube-api-access-x8zjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:20:32.212485 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:32.212459    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842" pod "ddf0f043-7884-4364-b12a-241f33271ad3" (UID: "ddf0f043-7884-4364-b12a-241f33271ad3"). InnerVolumeSpecName "pvc-6058e330-2857-4586-813b-f67a75f51416". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 23:20:32.310152 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:32.310110    5060 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" "
Oct 30 23:20:32.310152 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:32.310148    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zjx\" (UniqueName: \"kubernetes.io/projected/ddf0f043-7884-4364-b12a-241f33271ad3-kube-api-access-x8zjx\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:20:32.947013 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:32.946980    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"
Oct 30 23:20:32.947154 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:32.947026    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf" event={"ID":"ddf0f043-7884-4364-b12a-241f33271ad3","Type":"ContainerDied","Data":"d0bec6f7cc97d29feaa818d7a8e58dd8d00c7f67b39d0dd28c26c2da123e568e"}
Oct 30 23:20:32.947154 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:32.947064    5060 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0bec6f7cc97d29feaa818d7a8e58dd8d00c7f67b39d0dd28c26c2da123e568e"
Oct 30 23:20:33.283272 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:33.283247    5060 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6058e330-2857-4586-813b-f67a75f51416" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842") on node "ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 23:20:33.320138 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:33.320098    5060 reconciler_common.go:299] "Volume detached for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"csi-2938a377e19fedd411ba0532e0de3055c0d20a345c444844e77857b488b6d102\""
Oct 30 23:20:34.775546 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:20:34.775495    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:20:42.665922 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.665890    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"]
Oct 30 23:20:42.666466 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.666426    5060 cpu_manager.go:414] "containerMap: removing container" podUID="ddf0f043-7884-4364-b12a-241f33271ad3" containerName="storage-initializer"
Oct 30 23:20:42.666466 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.666446    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf0f043-7884-4364-b12a-241f33271ad3" containerName="storage-initializer"
Oct 30 23:20:42.666617 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.666556    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf0f043-7884-4364-b12a-241f33271ad3" containerName="storage-initializer"
Oct 30 23:20:42.671172 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.671148    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.682405 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.682377    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Oct 30 23:20:42.682548 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.682430    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Oct 30 23:20:42.682548 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.682433    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:20:42.682548 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.682456    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-mtvb8\""
Oct 30 23:20:42.682548 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.682466    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Oct 30 23:20:42.682548 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.682478    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 30 23:20:42.688273 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.688248    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"]
Oct 30 23:20:42.697743 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.697711    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.697875 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.697814    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.697875 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.697836    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.697875 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.697868    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.697994 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.697918    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.697994 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.697943    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-home\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.698109 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.698040    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-dshm\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.799352 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.799307    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-dshm\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.799565 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.799367    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.799565 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.799415    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.799565 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.799433    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.799565 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.799462    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctwkh\" (UniqueName: \"kubernetes.io/projected/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kube-api-access-ctwkh\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.799565 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.799485    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.799565 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.799535    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.799565 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.799563    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-home\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.799951 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.799892    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.800142 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.800047    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.800142 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.800069    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.800142 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.800104    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-home\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.800603 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.800586    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.801747 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.801724    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-dshm\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.802170 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.802151    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.900422 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.900380    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctwkh\" (UniqueName: \"kubernetes.io/projected/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kube-api-access-ctwkh\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:42.914381 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:42.914346    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctwkh\" (UniqueName: \"kubernetes.io/projected/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kube-api-access-ctwkh\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:45.117106 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:45.117061    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:45.124604 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:45.124573    5060 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:45.217973 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:45.217932    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:45.218154 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:45.218056    5060 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:45.222296 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:45.222275    5060 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") DevicePath \"csi-2938a377e19fedd411ba0532e0de3055c0d20a345c444844e77857b488b6d102\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:45.332223 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:45.332185    5060 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a5028c51e1c80b1a87f02ed2b08bcca484e56aa9640f5321f7da4c8d94842e15/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:45.375156 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:45.375068    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:45.387424 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:45.387385    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:45.674566 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:45.674534    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"]
Oct 30 23:20:45.676061 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:20:45.676028    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afcc2ff_a897_4dd7_88d9_08a8c22a1d3b.slice/crio-93207ccce69b4f86854b72e9c93c47b2d9d5f70a681c15129424b613a92f76b3 WatchSource:0}: Error finding container 93207ccce69b4f86854b72e9c93c47b2d9d5f70a681c15129424b613a92f76b3: Status 404 returned error can't find the container with id 93207ccce69b4f86854b72e9c93c47b2d9d5f70a681c15129424b613a92f76b3
Oct 30 23:20:46.006469 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:46.006430    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" event={"ID":"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b","Type":"ContainerStarted","Data":"909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36"}
Oct 30 23:20:46.006661 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:46.006493    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" event={"ID":"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b","Type":"ContainerStarted","Data":"93207ccce69b4f86854b72e9c93c47b2d9d5f70a681c15129424b613a92f76b3"}
Oct 30 23:20:49.021954 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:49.021918    5060 generic.go:356] "Generic (PLEG): container finished" podID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerID="909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36" exitCode=0
Oct 30 23:20:49.022361 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:49.021994    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" event={"ID":"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b","Type":"ContainerDied","Data":"909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36"}
Oct 30 23:20:50.028811 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:50.028773    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" event={"ID":"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b","Type":"ContainerStarted","Data":"dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96"}
Oct 30 23:20:50.388384 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:50.388288    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:55.206024 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.205955    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" podStartSLOduration=10.500834103999999 podStartE2EDuration="13.205932831s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.705098727s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:50.063851056 +0000 UTC m=+3730.883588876" watchObservedRunningTime="2026-10-30 23:20:55.205932831 +0000 UTC m=+3736.025670655"
Oct 30 23:20:55.208903 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.208860    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"]
Oct 30 23:20:55.214628 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.214602    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.231188 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.231155    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"]
Oct 30 23:20:55.236441 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.236409    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"]
Oct 30 23:20:55.236594 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.236569    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.250052 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.250010    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"]
Oct 30 23:20:55.327668 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.327626    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.327668 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.327667    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpdc\" (UniqueName: \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-kube-api-access-4mpdc\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.327929 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.327780    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.327929 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.327830    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.327929 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.327864    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.327929 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.327925    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.328095 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.327942    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.328095 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.327963    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.328095 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.328026    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.328095 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.328046    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.328241 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.328136    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.328241 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.328171    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.328241 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.328189    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940928d0-a858-47c6-a002-c05a252588db-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.328353 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.328268    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7f7a1f-6135-4eea-a257-787949cdaa82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.328353 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.328287    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.328353 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.328311    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.328353 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.328327    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940928d0-a858-47c6-a002-c05a252588db-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.328524 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.328418    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmcj8\" (UniqueName: \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-kube-api-access-nmcj8\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.387711 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.387667    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:20:55.389442 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.389408    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 30 23:20:55.429833 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.429798    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.429833 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.429839    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.430017 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.429856    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.430017 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.429873    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.430017 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.429889    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.430179 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430010    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.430179 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430058    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.430179 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430165    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.430329 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430238    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.430329 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430258    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.430329 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430286    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.430329 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430316    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940928d0-a858-47c6-a002-c05a252588db-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.430329 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430321    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.430597 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430348    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7f7a1f-6135-4eea-a257-787949cdaa82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.430597 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430379    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.430597 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430417    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.430597 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430431    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940928d0-a858-47c6-a002-c05a252588db-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.430597 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430481    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcj8\" (UniqueName: \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-kube-api-access-nmcj8\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.430597 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430540    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.430597 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430567    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpdc\" (UniqueName: \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-kube-api-access-4mpdc\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.430944 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430732    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.430944 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.430774    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.431052 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.431010    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.431197 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.431170    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7f7a1f-6135-4eea-a257-787949cdaa82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.431636 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.431604    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.431800 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.431642    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.431800 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.431683    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.432364 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.432338    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940928d0-a858-47c6-a002-c05a252588db-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.432627 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.432605    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.432882 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.432857    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.433309 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.433283    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.433650 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.433634    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940928d0-a858-47c6-a002-c05a252588db-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.448539 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.448479    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcj8\" (UniqueName: \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-kube-api-access-nmcj8\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.449334 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.449298    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.449473 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.449445    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-kxfn9\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.449766 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.449747    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpdc\" (UniqueName: \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-kube-api-access-4mpdc\") pod \"openshift-ai-inference-openshift-default-86b669b864-qjn66\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.529361 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.529261    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:55.560476 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.560442    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:55.813147 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.813117    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"]
Oct 30 23:20:55.816013 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:20:55.815983    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940928d0_a858_47c6_a002_c05a252588db.slice/crio-5512eee026512c6fe44c63bed7caa3d0b69e54a77ffd5d5c079abc8a308adeec WatchSource:0}: Error finding container 5512eee026512c6fe44c63bed7caa3d0b69e54a77ffd5d5c079abc8a308adeec: Status 404 returned error can't find the container with id 5512eee026512c6fe44c63bed7caa3d0b69e54a77ffd5d5c079abc8a308adeec
Oct 30 23:20:55.818345 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.818299    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:20:55.818431 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.818394    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:20:55.818470 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.818437    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:20:55.852575 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.852546    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"]
Oct 30 23:20:55.855007 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:20:55.854974    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7f7a1f_6135_4eea_a257_787949cdaa82.slice/crio-8b6b547dbd021f7de32cc0967d58b7481d84837dc4086914694dfaab2d8d1ba1 WatchSource:0}: Error finding container 8b6b547dbd021f7de32cc0967d58b7481d84837dc4086914694dfaab2d8d1ba1: Status 404 returned error can't find the container with id 8b6b547dbd021f7de32cc0967d58b7481d84837dc4086914694dfaab2d8d1ba1
Oct 30 23:20:55.857184 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.857149    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:20:55.857274 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.857215    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:20:55.857274 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:55.857248    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:20:56.060923 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.060878    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" event={"ID":"db7f7a1f-6135-4eea-a257-787949cdaa82","Type":"ContainerStarted","Data":"e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d"}
Oct 30 23:20:56.061090 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.060956    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" event={"ID":"db7f7a1f-6135-4eea-a257-787949cdaa82","Type":"ContainerStarted","Data":"8b6b547dbd021f7de32cc0967d58b7481d84837dc4086914694dfaab2d8d1ba1"}
Oct 30 23:20:56.063717 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.063161    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" event={"ID":"940928d0-a858-47c6-a002-c05a252588db","Type":"ContainerStarted","Data":"b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a"}
Oct 30 23:20:56.063717 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.063212    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" event={"ID":"940928d0-a858-47c6-a002-c05a252588db","Type":"ContainerStarted","Data":"5512eee026512c6fe44c63bed7caa3d0b69e54a77ffd5d5c079abc8a308adeec"}
Oct 30 23:20:56.085470 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.085407    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" podStartSLOduration=1.085376473 podStartE2EDuration="1.085376473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:56.084138836 +0000 UTC m=+3736.903876655" watchObservedRunningTime="2026-10-30 23:20:56.085376473 +0000 UTC m=+3736.905114293"
Oct 30 23:20:56.110063 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.110000    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" podStartSLOduration=1.109981377 podStartE2EDuration="1.109981377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:56.107672409 +0000 UTC m=+3736.927410232" watchObservedRunningTime="2026-10-30 23:20:56.109981377 +0000 UTC m=+3736.929719197"
Oct 30 23:20:56.530095 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.529865    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:20:56.531269 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.531243    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-kxfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:20:56.531362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.531287    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" podUID="940928d0-a858-47c6-a002-c05a252588db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 23:20:56.563050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.560907    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:20:56.563050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.562721    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-qjn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:20:56.563050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:56.562779    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 23:20:57.530924 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:57.530889    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-kxfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:20:57.531337 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:57.530942    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" podUID="940928d0-a858-47c6-a002-c05a252588db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 23:20:57.562006 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:57.561961    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-qjn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:20:57.562286 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:57.562259    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 23:20:58.530341 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:58.530303    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-kxfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:20:58.530528 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:58.530355    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" podUID="940928d0-a858-47c6-a002-c05a252588db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 23:20:58.560984 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:58.560937    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-qjn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:20:58.561627 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:58.561587    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 23:20:59.530421 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:59.530383    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-kxfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:20:59.530651 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:59.530452    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" podUID="940928d0-a858-47c6-a002-c05a252588db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 23:20:59.564532 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:59.560980    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-qjn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:20:59.564532 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:20:59.561047    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 23:21:00.530199 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:00.530163    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-kxfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:21:00.530436 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:00.530414    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" podUID="940928d0-a858-47c6-a002-c05a252588db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 23:21:00.561192 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:00.561144    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-qjn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:21:00.561498 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:00.561458    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 23:21:01.530010 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:01.529966    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-kxfn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 23:21:01.530651 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:01.530613    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" podUID="940928d0-a858-47c6-a002-c05a252588db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 23:21:01.561027 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:01.560984    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-qjn66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 23:21:01.561295 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:01.561267    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 23:21:02.534128 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:02.534092    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:21:02.534643 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:02.534484    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:21:02.535275 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:02.535253    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:21:02.564709 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:02.564670    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:21:02.564962 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:02.564944    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:21:02.565776 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:02.565756    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:21:04.776549 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:21:04.776492    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:21:05.388604 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:05.388560    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 30 23:21:15.388959 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:15.388906    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 30 23:21:25.388860 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:25.388807    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 30 23:21:34.777167 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:21:34.777123    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:21:35.388215 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:35.388170    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 30 23:21:45.388340 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:45.388291    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 30 23:21:55.388754 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:21:55.388708    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 30 23:22:04.777151 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:22:04.777112    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:22:05.397934 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:05.397894    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:22:05.405692 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:05.405658    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:22:30.334852 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:30.334775    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x_9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b/storage-initializer/0.log"
Oct 30 23:22:30.357627 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:30.357592    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x_9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b/main/0.log"
Oct 30 23:22:33.574386 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:33.574347    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"]
Oct 30 23:22:34.777053 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:22:34.776944    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:22:48.597931 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:48.597885    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main" containerID="cri-o://dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96" gracePeriod=45
Oct 30 23:22:49.241765 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.241728    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:22:49.259398 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259369    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-cabundle-cert\") pod \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") "
Oct 30 23:22:49.259545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259422    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tmp-dir\") pod \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") "
Oct 30 23:22:49.259545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259445    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-home\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-home\") pod \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") "
Oct 30 23:22:49.259545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259493    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tls-certs\") pod \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") "
Oct 30 23:22:49.259713 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259553    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kube-api-access-ctwkh\" (UniqueName: \"kubernetes.io/projected/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kube-api-access-ctwkh\") pod \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") "
Oct 30 23:22:49.259713 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259600    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-model-cache\") pod \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") "
Oct 30 23:22:49.259713 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259647    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kserve-provision-location\") pod \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") "
Oct 30 23:22:49.259713 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259682    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-dshm\") pod \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") "
Oct 30 23:22:49.259924 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259791    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") pod \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\" (UID: \"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b\") "
Oct 30 23:22:49.259924 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259837    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-cabundle-cert" pod "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" (UID: "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:22:49.259924 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.259859    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-home" pod "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" (UID: "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:49.260102 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.260080    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:49.260156 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.260109    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:49.260212 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.260173    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-model-cache" pod "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" (UID: "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:49.262074 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.262043    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kube-api-access-ctwkh" pod "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" (UID: "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b"). InnerVolumeSpecName "kube-api-access-ctwkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:49.262184 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.262167    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tls-certs" pod "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" (UID: "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:22:49.263853 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.263827    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-dshm" pod "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" (UID: "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:49.266411 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.266389    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842" pod "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" (UID: "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b"). InnerVolumeSpecName "pvc-6058e330-2857-4586-813b-f67a75f51416". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 23:22:49.273121 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.273090    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tmp-dir" pod "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" (UID: "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:49.286151 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.286117    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kserve-provision-location" pod "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" (UID: "9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:22:49.361472 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.361431    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:49.361677 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.361488    5060 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" "
Oct 30 23:22:49.361677 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.361520    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:49.361677 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.361535    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:49.361677 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.361549    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctwkh\" (UniqueName: \"kubernetes.io/projected/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kube-api-access-ctwkh\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:49.361677 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.361563    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:49.361677 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.361577    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:22:49.377385 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.377345    5060 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6058e330-2857-4586-813b-f67a75f51416" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842") on node "ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 23:22:49.462923 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.462884    5060 reconciler_common.go:299] "Volume detached for volume \"pvc-6058e330-2857-4586-813b-f67a75f51416\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b32c5fec5a9a5842\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"csi-2938a377e19fedd411ba0532e0de3055c0d20a345c444844e77857b488b6d102\""
Oct 30 23:22:49.643307 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.643215    5060 generic.go:356] "Generic (PLEG): container finished" podID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerID="dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96" exitCode=0
Oct 30 23:22:49.643307 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.643285    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"
Oct 30 23:22:49.643753 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.643299    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" event={"ID":"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b","Type":"ContainerDied","Data":"dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96"}
Oct 30 23:22:49.643753 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.643381    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x" event={"ID":"9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b","Type":"ContainerDied","Data":"93207ccce69b4f86854b72e9c93c47b2d9d5f70a681c15129424b613a92f76b3"}
Oct 30 23:22:49.643753 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.643410    5060 scope.go:122] "RemoveContainer" containerID="dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96"
Oct 30 23:22:49.652270 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.652246    5060 scope.go:122] "RemoveContainer" containerID="909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36"
Oct 30 23:22:49.659972 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.659954    5060 scope.go:122] "RemoveContainer" containerID="dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96"
Oct 30 23:22:49.660227 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:22:49.660208    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96\": container with ID starting with dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96 not found: ID does not exist" containerID="dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96"
Oct 30 23:22:49.660274 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.660237    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96"} err="failed to get container status \"dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96\": rpc error: code = NotFound desc = could not find container \"dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96\": container with ID starting with dbbe930ed4859e3f39ea26ae3f67aa81e50cf8aa3e1601a6ebde6885eae40e96 not found: ID does not exist"
Oct 30 23:22:49.660274 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.660255    5060 scope.go:122] "RemoveContainer" containerID="909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36"
Oct 30 23:22:49.660471 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:22:49.660457    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36\": container with ID starting with 909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36 not found: ID does not exist" containerID="909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36"
Oct 30 23:22:49.660529 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.660474    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36"} err="failed to get container status \"909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36\": rpc error: code = NotFound desc = could not find container \"909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36\": container with ID starting with 909b60043c74f0da1f961dac2a5a698255919d453aefb8d1726b475be60aef36 not found: ID does not exist"
Oct 30 23:22:49.664136 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.664114    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"]
Oct 30 23:22:49.670134 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:49.670112    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-86bf898848-8dp2x"]
Oct 30 23:22:51.289400 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:51.289363    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" path="/var/lib/kubelet/pods/9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b/volumes"
Oct 30 23:22:58.864490 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:58.864451    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"]
Oct 30 23:22:58.868714 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:58.868685    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-xvvcf"]
Oct 30 23:22:59.290800 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:22:59.290766    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf0f043-7884-4364-b12a-241f33271ad3" path="/var/lib/kubelet/pods/ddf0f043-7884-4364-b12a-241f33271ad3/volumes"
Oct 30 23:23:04.775430 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:23:04.775391    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:23:08.182108 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.182075    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"]
Oct 30 23:23:08.182651 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.182632    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="storage-initializer"
Oct 30 23:23:08.182739 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.182652    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="storage-initializer"
Oct 30 23:23:08.182739 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.182708    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main"
Oct 30 23:23:08.182739 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.182716    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main"
Oct 30 23:23:08.182896 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.182799    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afcc2ff-a897-4dd7-88d9-08a8c22a1d3b" containerName="main"
Oct 30 23:23:08.187880 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.187851    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.198906 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.198864    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Oct 30 23:23:08.199296 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.199241    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Oct 30 23:23:08.199385 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.199336    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-mnq7j\""
Oct 30 23:23:08.207321 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.207291    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"]
Oct 30 23:23:08.251091 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.250963    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:23:08.251091 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.250930    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Oct 30 23:23:08.251091 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.251026    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Oct 30 23:23:08.338695 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.338656    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2462f8-7501-4201-97fe-54962760298a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.338871 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.338710    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.338871 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.338777    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.338871 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.338838    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwpgp\" (UniqueName: \"kubernetes.io/projected/2d2462f8-7501-4201-97fe-54962760298a-kube-api-access-fwpgp\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.338974 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.338878    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d2462f8-7501-4201-97fe-54962760298a-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.338974 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.338906    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.339040 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.338985    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-home\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.339040 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.339018    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.440435 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.440341    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.440435 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.440390    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-home\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.440676 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.440534    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.440676 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.440624    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2462f8-7501-4201-97fe-54962760298a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.440750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.440692    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.440750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.440718    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.440852 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.440761    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpgp\" (UniqueName: \"kubernetes.io/projected/2d2462f8-7501-4201-97fe-54962760298a-kube-api-access-fwpgp\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.440852 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.440810    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d2462f8-7501-4201-97fe-54962760298a-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.440852 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.440844    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.441004 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.440878    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-home\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.441109 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.441088    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.441223 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.441201    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.441523 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.441488    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d2462f8-7501-4201-97fe-54962760298a-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.443125 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.443100    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.443265 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.443247    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2462f8-7501-4201-97fe-54962760298a-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.454889 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.454849    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpgp\" (UniqueName: \"kubernetes.io/projected/2d2462f8-7501-4201-97fe-54962760298a-kube-api-access-fwpgp\") pod \"router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.508183 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.508141    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:08.796690 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:08.796317    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"]
Oct 30 23:23:08.798646 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:23:08.798610    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2462f8_7501_4201_97fe_54962760298a.slice/crio-9112daf5bcc8240a7a4583f0d64343968607f74622c5f134b1efba7ea97b0a5d WatchSource:0}: Error finding container 9112daf5bcc8240a7a4583f0d64343968607f74622c5f134b1efba7ea97b0a5d: Status 404 returned error can't find the container with id 9112daf5bcc8240a7a4583f0d64343968607f74622c5f134b1efba7ea97b0a5d
Oct 30 23:23:09.736536 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:09.736468    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" event={"ID":"2d2462f8-7501-4201-97fe-54962760298a","Type":"ContainerStarted","Data":"96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2"}
Oct 30 23:23:09.736536 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:09.736543    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" event={"ID":"2d2462f8-7501-4201-97fe-54962760298a","Type":"ContainerStarted","Data":"9112daf5bcc8240a7a4583f0d64343968607f74622c5f134b1efba7ea97b0a5d"}
Oct 30 23:23:11.748588 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:11.748549    5060 generic.go:356] "Generic (PLEG): container finished" podID="2d2462f8-7501-4201-97fe-54962760298a" containerID="96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2" exitCode=0
Oct 30 23:23:11.749075 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:11.748626    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" event={"ID":"2d2462f8-7501-4201-97fe-54962760298a","Type":"ContainerDied","Data":"96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2"}
Oct 30 23:23:12.757375 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:12.757337    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" event={"ID":"2d2462f8-7501-4201-97fe-54962760298a","Type":"ContainerStarted","Data":"caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0"}
Oct 30 23:23:12.789949 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:12.789890    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podStartSLOduration=2.49128717 podStartE2EDuration="4.789874065s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.298586895s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:12.78592432 +0000 UTC m=+3873.605662165" watchObservedRunningTime="2026-10-30 23:23:12.789874065 +0000 UTC m=+3873.609611934"
Oct 30 23:23:18.509164 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:18.509122    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:18.509635 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:18.509190    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:23:18.511007 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:18.510977    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:23:23.070533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.069959    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"]
Oct 30 23:23:23.079636 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.079603    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.082047 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.081377    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"]
Oct 30 23:23:23.087604 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.087578    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Oct 30 23:23:23.087819 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.087749    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-x7k84\""
Oct 30 23:23:23.087892 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.087824    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Oct 30 23:23:23.088116 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.088097    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Oct 30 23:23:23.181318 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.181276    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-model-cache\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.181517 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.181329    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st9xr\" (UniqueName: \"kubernetes.io/projected/7d23da71-cc11-4a96-9e53-7aaf1571e2af-kube-api-access-st9xr\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.181517 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.181429    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tls-certs\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.181640 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.181572    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-dshm\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.181640 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.181605    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-home\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.181725 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.181635    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.282978 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.282926    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-model-cache\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.283170 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.283002    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st9xr\" (UniqueName: \"kubernetes.io/projected/7d23da71-cc11-4a96-9e53-7aaf1571e2af-kube-api-access-st9xr\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.283170 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.283050    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tls-certs\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.283170 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.283154    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-dshm\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.283346 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.283185    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-home\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.283346 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.283209    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.283449 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.283360    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-model-cache\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.283560 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.283539    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-home\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.283636 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.283615    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.285488 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.285465    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-dshm\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.285946 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.285922    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tls-certs\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:23.296321 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:23.296292    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st9xr\" (UniqueName: \"kubernetes.io/projected/7d23da71-cc11-4a96-9e53-7aaf1571e2af-kube-api-access-st9xr\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:27.829123 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:27.829069    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:27.837650 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:27.837605    5060 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:27.931098 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:27.931042    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:27.931492 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:27.931465    5060 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:27.935480 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:27.935445    5060 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") DevicePath \"csi-d51d045d772364b803ce2707a5cf0272674b7dd524ae831c96beafcba9482c96\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:28.237650 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:28.237613    5060 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8f49b9622264ac8e6b1761dc84d8d30508642bcf7a845f93c1f2f860e39897e4/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:28.333750 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:28.333700    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:28.505806 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:28.505704    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:28.512700 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:28.512655    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:23:29.017647 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:29.017532    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"]
Oct 30 23:23:29.039803 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:23:29.039757    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d23da71_cc11_4a96_9e53_7aaf1571e2af.slice/crio-c40ead870645d6f998f30d660faad2458e227aa0b12d916e3ae88997a6de7d9a WatchSource:0}: Error finding container c40ead870645d6f998f30d660faad2458e227aa0b12d916e3ae88997a6de7d9a: Status 404 returned error can't find the container with id c40ead870645d6f998f30d660faad2458e227aa0b12d916e3ae88997a6de7d9a
Oct 30 23:23:29.950359 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:29.950327    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" event={"ID":"7d23da71-cc11-4a96-9e53-7aaf1571e2af","Type":"ContainerStarted","Data":"3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85"}
Oct 30 23:23:29.950570 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:29.950382    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" event={"ID":"7d23da71-cc11-4a96-9e53-7aaf1571e2af","Type":"ContainerStarted","Data":"c40ead870645d6f998f30d660faad2458e227aa0b12d916e3ae88997a6de7d9a"}
Oct 30 23:23:29.978209 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:29.978150    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" podStartSLOduration=6.978135794 podStartE2EDuration="6.978135794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:29.976580691 +0000 UTC m=+3890.796318508" watchObservedRunningTime="2026-10-30 23:23:29.978135794 +0000 UTC m=+3890.797873613"
Oct 30 23:23:30.506523 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:30.506462    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:34.781548 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:23:34.781476    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:23:38.506853 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:38.506810    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:23:38.508381 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:38.508332    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 30 23:23:38.508568 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:38.508543    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:23:39.781670 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:39.781636    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:23:39.802735 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:39.802705    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:23:48.506879 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:48.506830    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 30 23:23:48.509200 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:48.509167    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:23:58.507182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:58.507061    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 30 23:23:58.509390 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:23:58.509356    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:24:04.778191 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:24:04.778146    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:24:08.507421 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:08.507366    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 30 23:24:08.508711 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:08.508678    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:24:18.506905 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:18.506856    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 30 23:24:18.509312 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:18.509267    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:24:28.506427 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:28.506383    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 30 23:24:28.508741 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:28.508698    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:24:34.779643 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:24:34.779596    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:24:38.507050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:38.506995    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 30 23:24:38.509406 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:38.509364    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:24:48.508913 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:48.508869    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:24:48.516636 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:48.516608    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:24:48.525164 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:48.525132    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:24:58.508557 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:24:58.508488    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:25:04.775981 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:25:04.775941    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:25:08.509227 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:08.509179    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:25:18.509221 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:18.509177    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Oct 30 23:25:28.518597 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:28.518490    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:25:28.526767 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:28.526721    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:25:34.783587 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:25:34.783540    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:25:35.569834 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:35.569803    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf_7d23da71-cc11-4a96-9e53-7aaf1571e2af/main/0.log"
Oct 30 23:25:38.914260 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:38.914221    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"]
Oct 30 23:25:40.455330 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:40.455291    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"]
Oct 30 23:25:53.940121 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:53.940061    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main" containerID="cri-o://3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85" gracePeriod=45
Oct 30 23:25:54.591000 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.590971    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:25:54.607906 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.607864    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d23da71-cc11-4a96-9e53-7aaf1571e2af-kube-api-access-st9xr\" (UniqueName: \"kubernetes.io/projected/7d23da71-cc11-4a96-9e53-7aaf1571e2af-kube-api-access-st9xr\") pod \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") "
Oct 30 23:25:54.608101 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.607913    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-home\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-home\") pod \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") "
Oct 30 23:25:54.608101 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.607940    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-model-cache\") pod \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") "
Oct 30 23:25:54.608101 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.607988    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tmp-dir\") pod \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") "
Oct 30 23:25:54.608269 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.608108    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\") pod \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") "
Oct 30 23:25:54.608269 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.608143    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tls-certs\") pod \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") "
Oct 30 23:25:54.608269 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.608204    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-dshm\") pod \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\" (UID: \"7d23da71-cc11-4a96-9e53-7aaf1571e2af\") "
Oct 30 23:25:54.608420 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.608258    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-model-cache" pod "7d23da71-cc11-4a96-9e53-7aaf1571e2af" (UID: "7d23da71-cc11-4a96-9e53-7aaf1571e2af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:25:54.608420 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.608391    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-home" pod "7d23da71-cc11-4a96-9e53-7aaf1571e2af" (UID: "7d23da71-cc11-4a96-9e53-7aaf1571e2af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:25:54.608549 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.608530    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:54.608622 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.608555    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:54.610776 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.610705    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tls-certs" pod "7d23da71-cc11-4a96-9e53-7aaf1571e2af" (UID: "7d23da71-cc11-4a96-9e53-7aaf1571e2af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:25:54.610776 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.610729    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d23da71-cc11-4a96-9e53-7aaf1571e2af-kube-api-access-st9xr" pod "7d23da71-cc11-4a96-9e53-7aaf1571e2af" (UID: "7d23da71-cc11-4a96-9e53-7aaf1571e2af"). InnerVolumeSpecName "kube-api-access-st9xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:25:54.610928 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.610774    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-dshm" pod "7d23da71-cc11-4a96-9e53-7aaf1571e2af" (UID: "7d23da71-cc11-4a96-9e53-7aaf1571e2af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:25:54.616595 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.616569    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66" pod "7d23da71-cc11-4a96-9e53-7aaf1571e2af" (UID: "7d23da71-cc11-4a96-9e53-7aaf1571e2af"). InnerVolumeSpecName "pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 23:25:54.621187 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.621159    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tmp-dir" pod "7d23da71-cc11-4a96-9e53-7aaf1571e2af" (UID: "7d23da71-cc11-4a96-9e53-7aaf1571e2af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:25:54.689827 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.689739    5060 generic.go:356] "Generic (PLEG): container finished" podID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerID="3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85" exitCode=0
Oct 30 23:25:54.689827 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.689791    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"
Oct 30 23:25:54.690011 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.689826    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" event={"ID":"7d23da71-cc11-4a96-9e53-7aaf1571e2af","Type":"ContainerDied","Data":"3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85"}
Oct 30 23:25:54.690011 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.689885    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" event={"ID":"7d23da71-cc11-4a96-9e53-7aaf1571e2af","Type":"ContainerDied","Data":"c40ead870645d6f998f30d660faad2458e227aa0b12d916e3ae88997a6de7d9a"}
Oct 30 23:25:54.690011 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.689907    5060 scope.go:122] "RemoveContainer" containerID="3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85"
Oct 30 23:25:54.699317 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.699297    5060 scope.go:122] "RemoveContainer" containerID="3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85"
Oct 30 23:25:54.699622 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:25:54.699602    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85\": container with ID starting with 3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85 not found: ID does not exist" containerID="3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85"
Oct 30 23:25:54.699698 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.699630    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85"} err="failed to get container status \"3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85\": rpc error: code = NotFound desc = could not find container \"3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85\": container with ID starting with 3e848635fbba4e837dffe8ddb46ba62ab082fa725774d4e1f1248d832a240d85 not found: ID does not exist"
Oct 30 23:25:54.709207 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.709179    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:54.709315 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.709226    5060 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" "
Oct 30 23:25:54.709315 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.709236    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d23da71-cc11-4a96-9e53-7aaf1571e2af-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:54.709315 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.709245    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d23da71-cc11-4a96-9e53-7aaf1571e2af-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:54.709315 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.709254    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st9xr\" (UniqueName: \"kubernetes.io/projected/7d23da71-cc11-4a96-9e53-7aaf1571e2af-kube-api-access-st9xr\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:54.723291 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.723255    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"]
Oct 30 23:25:54.728711 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.728680    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf"]
Oct 30 23:25:54.738949 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.738925    5060 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66") on node "ip-10-0-138-251.us-east-2.compute.internal"
Oct 30 23:25:54.809732 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:54.809692    5060 reconciler_common.go:299] "Volume detached for volume \"pvc-f1c249b5-b7c9-4e8a-ade6-5bed0805d5ad\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc959ddd15c69d66\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"csi-d51d045d772364b803ce2707a5cf0272674b7dd524ae831c96beafcba9482c96\""
Oct 30 23:25:55.290697 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:55.290663    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" path="/var/lib/kubelet/pods/7d23da71-cc11-4a96-9e53-7aaf1571e2af/volumes"
Oct 30 23:25:55.478217 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:55.478171    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main" containerID="cri-o://caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0" gracePeriod=45
Oct 30 23:25:55.517984 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:55.517935    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-r6xjf" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded"
Oct 30 23:25:56.230185 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.230155    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:25:56.422477 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.422388    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2462f8-7501-4201-97fe-54962760298a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2462f8-7501-4201-97fe-54962760298a-tls-certs\") pod \"2d2462f8-7501-4201-97fe-54962760298a\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") "
Oct 30 23:25:56.422477 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.422452    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2462f8-7501-4201-97fe-54962760298a-kube-api-access-fwpgp\" (UniqueName: \"kubernetes.io/projected/2d2462f8-7501-4201-97fe-54962760298a-kube-api-access-fwpgp\") pod \"2d2462f8-7501-4201-97fe-54962760298a\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") "
Oct 30 23:25:56.422477 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.422473    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-kserve-provision-location\") pod \"2d2462f8-7501-4201-97fe-54962760298a\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") "
Oct 30 23:25:56.423090 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.422501    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-home\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-home\") pod \"2d2462f8-7501-4201-97fe-54962760298a\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") "
Oct 30 23:25:56.423090 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.422608    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-model-cache\") pod \"2d2462f8-7501-4201-97fe-54962760298a\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") "
Oct 30 23:25:56.423090 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.422692    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-tmp-dir\") pod \"2d2462f8-7501-4201-97fe-54962760298a\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") "
Oct 30 23:25:56.423090 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.422725    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-dshm\") pod \"2d2462f8-7501-4201-97fe-54962760298a\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") "
Oct 30 23:25:56.423090 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.422751    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2462f8-7501-4201-97fe-54962760298a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d2462f8-7501-4201-97fe-54962760298a-cabundle-cert\") pod \"2d2462f8-7501-4201-97fe-54962760298a\" (UID: \"2d2462f8-7501-4201-97fe-54962760298a\") "
Oct 30 23:25:56.423090 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.422874    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-model-cache" pod "2d2462f8-7501-4201-97fe-54962760298a" (UID: "2d2462f8-7501-4201-97fe-54962760298a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:25:56.423090 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.423035    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:56.423446 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.423189    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2462f8-7501-4201-97fe-54962760298a-cabundle-cert" pod "2d2462f8-7501-4201-97fe-54962760298a" (UID: "2d2462f8-7501-4201-97fe-54962760298a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:25:56.423503 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.423471    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-home" pod "2d2462f8-7501-4201-97fe-54962760298a" (UID: "2d2462f8-7501-4201-97fe-54962760298a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:25:56.424967 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.424932    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2462f8-7501-4201-97fe-54962760298a-tls-certs" pod "2d2462f8-7501-4201-97fe-54962760298a" (UID: "2d2462f8-7501-4201-97fe-54962760298a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:25:56.425097 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.425049    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2462f8-7501-4201-97fe-54962760298a-kube-api-access-fwpgp" pod "2d2462f8-7501-4201-97fe-54962760298a" (UID: "2d2462f8-7501-4201-97fe-54962760298a"). InnerVolumeSpecName "kube-api-access-fwpgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:25:56.425396 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.425369    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-dshm" pod "2d2462f8-7501-4201-97fe-54962760298a" (UID: "2d2462f8-7501-4201-97fe-54962760298a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:25:56.442634 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.434530    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-tmp-dir" pod "2d2462f8-7501-4201-97fe-54962760298a" (UID: "2d2462f8-7501-4201-97fe-54962760298a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:25:56.450468 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.450431    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-kserve-provision-location" pod "2d2462f8-7501-4201-97fe-54962760298a" (UID: "2d2462f8-7501-4201-97fe-54962760298a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:25:56.523477 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.523432    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwpgp\" (UniqueName: \"kubernetes.io/projected/2d2462f8-7501-4201-97fe-54962760298a-kube-api-access-fwpgp\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:56.523477 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.523469    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:56.523477 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.523481    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:56.523753 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.523541    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:56.523753 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.523552    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2462f8-7501-4201-97fe-54962760298a-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:56.523753 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.523560    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2d2462f8-7501-4201-97fe-54962760298a-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:56.523753 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.523569    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2462f8-7501-4201-97fe-54962760298a-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:25:56.700826 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.700792    5060 generic.go:356] "Generic (PLEG): container finished" podID="2d2462f8-7501-4201-97fe-54962760298a" containerID="caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0" exitCode=0
Oct 30 23:25:56.701002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.700854    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"
Oct 30 23:25:56.701002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.700873    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" event={"ID":"2d2462f8-7501-4201-97fe-54962760298a","Type":"ContainerDied","Data":"caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0"}
Oct 30 23:25:56.701002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.700948    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4" event={"ID":"2d2462f8-7501-4201-97fe-54962760298a","Type":"ContainerDied","Data":"9112daf5bcc8240a7a4583f0d64343968607f74622c5f134b1efba7ea97b0a5d"}
Oct 30 23:25:56.701002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.700970    5060 scope.go:122] "RemoveContainer" containerID="caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0"
Oct 30 23:25:56.710898 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.710877    5060 scope.go:122] "RemoveContainer" containerID="96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2"
Oct 30 23:25:56.720987 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.720959    5060 scope.go:122] "RemoveContainer" containerID="caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0"
Oct 30 23:25:56.721431 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:25:56.721405    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0\": container with ID starting with caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0 not found: ID does not exist" containerID="caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0"
Oct 30 23:25:56.721546 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.721441    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0"} err="failed to get container status \"caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0\": rpc error: code = NotFound desc = could not find container \"caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0\": container with ID starting with caa8775b4a606b1c4366e482f5f7c6c2f7600b73d88a2f1e1f5f616b926d56b0 not found: ID does not exist"
Oct 30 23:25:56.721546 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.721461    5060 scope.go:122] "RemoveContainer" containerID="96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2"
Oct 30 23:25:56.721748 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:25:56.721730    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2\": container with ID starting with 96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2 not found: ID does not exist" containerID="96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2"
Oct 30 23:25:56.721793 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.721753    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2"} err="failed to get container status \"96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2\": rpc error: code = NotFound desc = could not find container \"96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2\": container with ID starting with 96cc858e01222b5899a56159d4fad0cdc79acb03042e7e89c578eb717514f9a2 not found: ID does not exist"
Oct 30 23:25:56.727041 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.727017    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"]
Oct 30 23:25:56.731394 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:56.731372    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-59bb995bd9-kjgp4"]
Oct 30 23:25:57.288937 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:57.288903    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2462f8-7501-4201-97fe-54962760298a" path="/var/lib/kubelet/pods/2d2462f8-7501-4201-97fe-54962760298a/volumes"
Oct 30 23:25:59.770419 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.770385    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"]
Oct 30 23:25:59.771384 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.771353    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="storage-initializer"
Oct 30 23:25:59.771384 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.771380    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="storage-initializer"
Oct 30 23:25:59.771577 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.771397    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main"
Oct 30 23:25:59.771577 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.771404    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main"
Oct 30 23:25:59.771577 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.771426    5060 cpu_manager.go:414] "containerMap: removing container" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main"
Oct 30 23:25:59.771577 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.771431    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main"
Oct 30 23:25:59.771577 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.771489    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2462f8-7501-4201-97fe-54962760298a" containerName="main"
Oct 30 23:25:59.771577 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.771500    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d23da71-cc11-4a96-9e53-7aaf1571e2af" containerName="main"
Oct 30 23:25:59.775006 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.774986    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.779982 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.779959    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"model-routing-only-gateway-openshift-default-dockercfg-djb5j\""
Oct 30 23:25:59.780545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.780500    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"openshift-service-ca.crt\""
Oct 30 23:25:59.780676 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.780545    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"openshift-gw-ca-root-cert\""
Oct 30 23:25:59.780676 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.780658    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"kube-root-ca.crt\""
Oct 30 23:25:59.792767 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.792734    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"]
Oct 30 23:25:59.847599 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.847555    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.847599 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.847602    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.847874 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.847673    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-credential-socket\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.847874 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.847787    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-token\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.847874 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.847824    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-data\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.847874 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.847844    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb97217b-b202-4bb8-bf0d-b386f110d415-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.848078 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.847876    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv8q7\" (UniqueName: \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-kube-api-access-dv8q7\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.848078 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.847939    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-socket\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.848078 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.848028    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-certs\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.948979 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.948937    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-socket\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949194 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949009    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-certs\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949194 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949038    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949194 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949061    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949194 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949093    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-credential-socket\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949194 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949135    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-token\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949194 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949169    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-data\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949194 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949192    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb97217b-b202-4bb8-bf0d-b386f110d415-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949598 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949238    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv8q7\" (UniqueName: \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-kube-api-access-dv8q7\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949598 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949430    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-socket\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949598 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949562    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-certs\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949824 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949713    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-data\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.949943 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.949918    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-credential-socket\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.950112 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.950090    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb97217b-b202-4bb8-bf0d-b386f110d415-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.951342 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.951318    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.951625 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.951603    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.966116 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.966076    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv8q7\" (UniqueName: \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-kube-api-access-dv8q7\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:25:59.966882 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:25:59.966863    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-token\") pod \"model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:26:00.089719 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:00.089627    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:26:00.394360 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:00.394327    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"]
Oct 30 23:26:00.395102 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:26:00.395079    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb97217b_b202_4bb8_bf0d_b386f110d415.slice/crio-0d8b09d847742623347d17400be9ff8c790dc9090380671c44fcc7975ac4511e WatchSource:0}: Error finding container 0d8b09d847742623347d17400be9ff8c790dc9090380671c44fcc7975ac4511e: Status 404 returned error can't find the container with id 0d8b09d847742623347d17400be9ff8c790dc9090380671c44fcc7975ac4511e
Oct 30 23:26:00.397237 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:00.397201    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:26:00.397324 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:00.397276    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:26:00.397379 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:00.397319    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:26:00.722895 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:00.722852    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" event={"ID":"fb97217b-b202-4bb8-bf0d-b386f110d415","Type":"ContainerStarted","Data":"a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd"}
Oct 30 23:26:00.723044 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:00.722919    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" event={"ID":"fb97217b-b202-4bb8-bf0d-b386f110d415","Type":"ContainerStarted","Data":"0d8b09d847742623347d17400be9ff8c790dc9090380671c44fcc7975ac4511e"}
Oct 30 23:26:00.751576 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:00.751519    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" podStartSLOduration=1.751492835 podStartE2EDuration="1.751492835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:26:00.748857064 +0000 UTC m=+4041.568594881" watchObservedRunningTime="2026-10-30 23:26:00.751492835 +0000 UTC m=+4041.571230645"
Oct 30 23:26:01.090534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:01.090429    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:26:01.095573 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:01.095541    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:26:01.727844 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:01.727811    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:26:01.728817 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:01.728800    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:26:04.795952 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:26:04.795908    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:26:13.804443 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:13.804406    5060 generic.go:356] "Generic (PLEG): container finished" podID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerID="a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd" exitCode=0
Oct 30 23:26:13.804877 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:13.804477    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" event={"ID":"fb97217b-b202-4bb8-bf0d-b386f110d415","Type":"ContainerDied","Data":"a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd"}
Oct 30 23:26:13.804877 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:13.804870    5060 scope.go:122] "RemoveContainer" containerID="a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd"
Oct 30 23:26:13.805916 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:13.805886    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:26:13.805989 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:13.805971    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:26:13.806037 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:13.806021    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:26:14.811104 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:14.811068    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" event={"ID":"fb97217b-b202-4bb8-bf0d-b386f110d415","Type":"ContainerStarted","Data":"c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f"}
Oct 30 23:26:15.090277 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:15.090180    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:26:15.091381 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:15.091351    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 23:26:16.090228 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:16.090188    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 23:26:16.728445 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:16.728406    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:26:17.090886 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:17.090793    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 23:26:18.090050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:18.090004    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 23:26:19.090478 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:19.090427    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 23:26:20.094441 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:20.094405    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:26:20.095474 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:20.095453    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:26:34.775704 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:26:34.775664    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:26:40.184317 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:26:40.184284    5060 scope.go:122] "RemoveContainer" containerID="4490a433b1f80a57b4eac392aaa834dfe4c4a450689a88c349a20edd8b6f5c76"
Oct 30 23:27:04.774954 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:27:04.774912    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:27:19.989674 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:19.989634    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"]
Oct 30 23:27:19.993224 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:19.993196    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:19.999277 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:19.999245    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"openshift-service-ca.crt\""
Oct 30 23:27:20.000189 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.000165    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"kube-root-ca.crt\""
Oct 30 23:27:20.000328 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.000252    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"model-routing-only-7aad4d71-kserve-self-signed-certs\""
Oct 30 23:27:20.000442 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.000427    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"default-dockercfg-p7nk4\""
Oct 30 23:27:20.008571 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.007959    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"]
Oct 30 23:27:20.060255 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.060216    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-model-cache\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.060432 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.060270    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvmv\" (UniqueName: \"kubernetes.io/projected/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-kube-api-access-jpvmv\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.060432 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.060346    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tmp-dir\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.060432 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.060411    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-dshm\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.060583 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.060479    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tls-certs\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.060583 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.060528    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-home\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.161426 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.161390    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-dshm\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.161426 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.161436    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tls-certs\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.161752 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.161454    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-home\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.161752 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.161493    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-model-cache\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.161752 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.161559    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvmv\" (UniqueName: \"kubernetes.io/projected/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-kube-api-access-jpvmv\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.161752 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.161586    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tmp-dir\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.161998 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.161972    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-model-cache\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.161998 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.161992    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tmp-dir\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.162102 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.162032    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-home\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.163860 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.163839    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-dshm\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.164055 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.164037    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tls-certs\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.174542 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.174517    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvmv\" (UniqueName: \"kubernetes.io/projected/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-kube-api-access-jpvmv\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.315365 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.315275    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:20.595724 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:20.595675    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"]
Oct 30 23:27:20.598160 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:27:20.598134    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f7e4f4_1a4d_4bca_8cc0_a0e50c9c78dd.slice/crio-b1edb3a2447e2f5b38f477855cf87d7273ed6177fcf2d30cb9d83f13ec045da9 WatchSource:0}: Error finding container b1edb3a2447e2f5b38f477855cf87d7273ed6177fcf2d30cb9d83f13ec045da9: Status 404 returned error can't find the container with id b1edb3a2447e2f5b38f477855cf87d7273ed6177fcf2d30cb9d83f13ec045da9
Oct 30 23:27:21.108898 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:21.108860    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn" event={"ID":"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd","Type":"ContainerStarted","Data":"51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f"}
Oct 30 23:27:21.109301 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:21.108911    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn" event={"ID":"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd","Type":"ContainerStarted","Data":"b1edb3a2447e2f5b38f477855cf87d7273ed6177fcf2d30cb9d83f13ec045da9"}
Oct 30 23:27:21.132558 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:21.132479    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn" podStartSLOduration=2.132465102 podStartE2EDuration="2.132465102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:21.13192746 +0000 UTC m=+4121.951665279" watchObservedRunningTime="2026-10-30 23:27:21.132465102 +0000 UTC m=+4121.952202974"
Oct 30 23:27:21.138725 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:21.138695    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"]
Oct 30 23:27:21.139033 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:21.138985    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy" containerID="cri-o://c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f" gracePeriod=30
Oct 30 23:27:21.316320 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:21.316280    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:26.382784 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.382754    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:27:26.523722 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.523680    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-certs\") pod \"fb97217b-b202-4bb8-bf0d-b386f110d415\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") "
Oct 30 23:27:26.523918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.523788    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-credential-socket\") pod \"fb97217b-b202-4bb8-bf0d-b386f110d415\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") "
Oct 30 23:27:26.523918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.523822    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-data\") pod \"fb97217b-b202-4bb8-bf0d-b386f110d415\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") "
Oct 30 23:27:26.523918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.523851    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-podinfo\") pod \"fb97217b-b202-4bb8-bf0d-b386f110d415\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") "
Oct 30 23:27:26.523918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.523886    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-token\" (UniqueName: \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-token\") pod \"fb97217b-b202-4bb8-bf0d-b386f110d415\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") "
Oct 30 23:27:26.524134 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.523930    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-socket\") pod \"fb97217b-b202-4bb8-bf0d-b386f110d415\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") "
Oct 30 23:27:26.524134 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.523984    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-certs" pod "fb97217b-b202-4bb8-bf0d-b386f110d415" (UID: "fb97217b-b202-4bb8-bf0d-b386f110d415"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:27:26.524134 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.523989    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-kube-api-access-dv8q7\" (UniqueName: \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-kube-api-access-dv8q7\") pod \"fb97217b-b202-4bb8-bf0d-b386f110d415\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") "
Oct 30 23:27:26.524134 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.524108    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb97217b-b202-4bb8-bf0d-b386f110d415-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb97217b-b202-4bb8-bf0d-b386f110d415-istiod-ca-cert\") pod \"fb97217b-b202-4bb8-bf0d-b386f110d415\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") "
Oct 30 23:27:26.524343 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.524176    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-envoy\") pod \"fb97217b-b202-4bb8-bf0d-b386f110d415\" (UID: \"fb97217b-b202-4bb8-bf0d-b386f110d415\") "
Oct 30 23:27:26.524792 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.524487    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-socket" pod "fb97217b-b202-4bb8-bf0d-b386f110d415" (UID: "fb97217b-b202-4bb8-bf0d-b386f110d415"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:27:26.524792 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.524680    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:27:26.524792 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.524703    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:27:26.524792 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.524764    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb97217b-b202-4bb8-bf0d-b386f110d415-istiod-ca-cert" pod "fb97217b-b202-4bb8-bf0d-b386f110d415" (UID: "fb97217b-b202-4bb8-bf0d-b386f110d415"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:27:26.525800 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.525770    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-credential-socket" pod "fb97217b-b202-4bb8-bf0d-b386f110d415" (UID: "fb97217b-b202-4bb8-bf0d-b386f110d415"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:27:26.526035 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.526012    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-data" pod "fb97217b-b202-4bb8-bf0d-b386f110d415" (UID: "fb97217b-b202-4bb8-bf0d-b386f110d415"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:27:26.526784 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.526757    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-kube-api-access-dv8q7" pod "fb97217b-b202-4bb8-bf0d-b386f110d415" (UID: "fb97217b-b202-4bb8-bf0d-b386f110d415"). InnerVolumeSpecName "kube-api-access-dv8q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:27:26.526890 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.526788    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-podinfo" pod "fb97217b-b202-4bb8-bf0d-b386f110d415" (UID: "fb97217b-b202-4bb8-bf0d-b386f110d415"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:27:26.527106 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.527092    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-envoy" pod "fb97217b-b202-4bb8-bf0d-b386f110d415" (UID: "fb97217b-b202-4bb8-bf0d-b386f110d415"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:27:26.527489 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.527474    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-token" pod "fb97217b-b202-4bb8-bf0d-b386f110d415" (UID: "fb97217b-b202-4bb8-bf0d-b386f110d415"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:27:26.625394 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.625355    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb97217b-b202-4bb8-bf0d-b386f110d415-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:27:26.625394 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.625387    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:27:26.625394 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.625398    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:27:26.625644 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.625407    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:27:26.625644 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.625416    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:27:26.625644 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.625424    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:27:26.625644 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:26.625433    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv8q7\" (UniqueName: \"kubernetes.io/projected/fb97217b-b202-4bb8-bf0d-b386f110d415-kube-api-access-dv8q7\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:27:27.138388 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.138354    5060 generic.go:356] "Generic (PLEG): container finished" podID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerID="c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f" exitCode=0
Oct 30 23:27:27.138589 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.138401    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"
Oct 30 23:27:27.138589 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.138429    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" event={"ID":"fb97217b-b202-4bb8-bf0d-b386f110d415","Type":"ContainerDied","Data":"c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f"}
Oct 30 23:27:27.138589 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.138484    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv" event={"ID":"fb97217b-b202-4bb8-bf0d-b386f110d415","Type":"ContainerDied","Data":"0d8b09d847742623347d17400be9ff8c790dc9090380671c44fcc7975ac4511e"}
Oct 30 23:27:27.138589 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.138520    5060 scope.go:122] "RemoveContainer" containerID="c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f"
Oct 30 23:27:27.148547 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.148528    5060 scope.go:122] "RemoveContainer" containerID="a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd"
Oct 30 23:27:27.160199 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.160166    5060 scope.go:122] "RemoveContainer" containerID="c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f"
Oct 30 23:27:27.160653 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:27:27.160624    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f\": container with ID starting with c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f not found: ID does not exist" containerID="c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f"
Oct 30 23:27:27.160740 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.160655    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f"} err="failed to get container status \"c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f\": rpc error: code = NotFound desc = could not find container \"c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f\": container with ID starting with c584fef057351b579d1a484b4bbcd3a4577e391342d7a02ff4890635b738dc0f not found: ID does not exist"
Oct 30 23:27:27.160740 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.160672    5060 scope.go:122] "RemoveContainer" containerID="a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd"
Oct 30 23:27:27.160952 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:27:27.160938    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd\": container with ID starting with a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd not found: ID does not exist" containerID="a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd"
Oct 30 23:27:27.161000 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.160959    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd"} err="failed to get container status \"a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd\": rpc error: code = NotFound desc = could not find container \"a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd\": container with ID starting with a96cb391553604a1577e061e481a1138cd2ceea1e91882e0f7a8e53f73d6e1dd not found: ID does not exist"
Oct 30 23:27:27.165428 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.165402    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"]
Oct 30 23:27:27.171907 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.171888    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5b4cd574df-cs7hv"]
Oct 30 23:27:27.290021 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:27.289985    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" path="/var/lib/kubelet/pods/fb97217b-b202-4bb8-bf0d-b386f110d415/volumes"
Oct 30 23:27:30.315941 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:30.315903    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:30.328118 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:30.328083    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:31.167887 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:27:31.167854    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:27:34.775026 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:27:34.774987    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:28:04.775823 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:28:04.775787    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:28:34.775705 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:28:34.775622    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:28:37.907796 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:37.907756    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"]
Oct 30 23:28:37.917414 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:28:37.917370    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f" cmd=["/bin/sleep","15"]
Oct 30 23:28:37.917591 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:37.917425    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 23:28:37.917591 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:28:37.917439    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn" podUID="62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" containerName="main" containerID="cri-o://51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f"
Oct 30 23:28:37.917591 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:37.917485    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn" podUID="62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" containerName="main" containerID="cri-o://51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f" gracePeriod=60
Oct 30 23:28:38.158033 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.157927    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn" podUID="62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Oct 30 23:28:38.264600 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.264570    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:28:38.351802 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.351766    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-dshm\") pod \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") "
Oct 30 23:28:38.352027 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.351816    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-kube-api-access-jpvmv\" (UniqueName: \"kubernetes.io/projected/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-kube-api-access-jpvmv\") pod \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") "
Oct 30 23:28:38.352027 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.351858    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-model-cache\") pod \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") "
Oct 30 23:28:38.352027 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.351900    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tmp-dir\") pod \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") "
Oct 30 23:28:38.352027 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.351950    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-home\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-home\") pod \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") "
Oct 30 23:28:38.352027 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.352008    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tls-certs\") pod \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\" (UID: \"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd\") "
Oct 30 23:28:38.352273 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.352176    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-model-cache" pod "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" (UID: "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:28:38.352273 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.352194    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tmp-dir" pod "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" (UID: "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:28:38.352273 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.352258    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-home" pod "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" (UID: "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:28:38.352434 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.352414    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.352474 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.352441    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.352474 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.352457    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.354100 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.354073    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-dshm" pod "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" (UID: "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:28:38.354209 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.354077    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-kube-api-access-jpvmv" pod "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" (UID: "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd"). InnerVolumeSpecName "kube-api-access-jpvmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:38.354209 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.354083    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tls-certs" pod "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" (UID: "62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:38.453312 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.453276    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.453312 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.453303    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvmv\" (UniqueName: \"kubernetes.io/projected/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-kube-api-access-jpvmv\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.453312 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.453313    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.457772 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.457737    5060 generic.go:356] "Generic (PLEG): container finished" podID="62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" containerID="51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f" exitCode=0
Oct 30 23:28:38.457900 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.457780    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"
Oct 30 23:28:38.457900 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.457811    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn" event={"ID":"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd","Type":"ContainerDied","Data":"51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f"}
Oct 30 23:28:38.457900 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.457856    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn" event={"ID":"62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd","Type":"ContainerDied","Data":"b1edb3a2447e2f5b38f477855cf87d7273ed6177fcf2d30cb9d83f13ec045da9"}
Oct 30 23:28:38.457900 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.457878    5060 scope.go:122] "RemoveContainer" containerID="51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f"
Oct 30 23:28:38.467279 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.467260    5060 scope.go:122] "RemoveContainer" containerID="51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f"
Oct 30 23:28:38.467571 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:28:38.467547    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f\": container with ID starting with 51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f not found: ID does not exist" containerID="51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f"
Oct 30 23:28:38.467622 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.467584    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f"} err="failed to get container status \"51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f\": rpc error: code = NotFound desc = could not find container \"51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f\": container with ID starting with 51871d43e10830423c9eb1378edc695f46a74a12d423abe4bedf949d8b523b2f not found: ID does not exist"
Oct 30 23:28:38.482429 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.482392    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"]
Oct 30 23:28:38.486747 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:38.486717    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-7aad4d71-kserve-6557db66dd-7fsnn"]
Oct 30 23:28:39.289649 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:39.289618    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" path="/var/lib/kubelet/pods/62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd/volumes"
Oct 30 23:28:39.811409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:39.811380    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:28:39.836973 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:39.836942    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:28:43.693565 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.693526    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"]
Oct 30 23:28:43.694083 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.694060    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy"
Oct 30 23:28:43.694162 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.694086    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy"
Oct 30 23:28:43.694162 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.694102    5060 cpu_manager.go:414] "containerMap: removing container" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy"
Oct 30 23:28:43.694162 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.694109    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy"
Oct 30 23:28:43.694300 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.694167    5060 cpu_manager.go:414] "containerMap: removing container" podUID="62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" containerName="main"
Oct 30 23:28:43.694300 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.694176    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" containerName="main"
Oct 30 23:28:43.694300 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.694266    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f7e4f4-1a4d-4bca-8cc0-a0e50c9c78dd" containerName="main"
Oct 30 23:28:43.694300 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.694281    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy"
Oct 30 23:28:43.694300 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.694291    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb97217b-b202-4bb8-bf0d-b386f110d415" containerName="istio-proxy"
Oct 30 23:28:43.698971 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.698945    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.703702 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.703672    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Oct 30 23:28:43.703702 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.703674    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:28:43.703866 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.703672    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Oct 30 23:28:43.704926 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.704904    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Oct 30 23:28:43.704926 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.704916    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Oct 30 23:28:43.705082 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.704927    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-8qw8n\""
Oct 30 23:28:43.711521 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.711450    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"]
Oct 30 23:28:43.798406 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.798360    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.798406 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.798406    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.798720 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.798532    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.798720 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.798587    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.798720 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.798622    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr7vn\" (UniqueName: \"kubernetes.io/projected/613946a6-253b-4b4c-906e-00a6faa17df8-kube-api-access-tr7vn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.798720 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.798664    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613946a6-253b-4b4c-906e-00a6faa17df8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.798878 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.798762    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.798878 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.798801    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/613946a6-253b-4b4c-906e-00a6faa17df8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.899628 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.899583    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613946a6-253b-4b4c-906e-00a6faa17df8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.899838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.899659    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.899838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.899692    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/613946a6-253b-4b4c-906e-00a6faa17df8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.899838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.899720    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.899838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.899748    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.899838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.899796    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.899838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.899828    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.900131 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.899857    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr7vn\" (UniqueName: \"kubernetes.io/projected/613946a6-253b-4b4c-906e-00a6faa17df8-kube-api-access-tr7vn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.900205 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.900159    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.900253 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.900222    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.900302 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.900253    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.900464 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.900446    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.900566 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.900550    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/613946a6-253b-4b4c-906e-00a6faa17df8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.902481 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.902462    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.902891 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.902868    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613946a6-253b-4b4c-906e-00a6faa17df8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:43.913462 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:43.913422    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr7vn\" (UniqueName: \"kubernetes.io/projected/613946a6-253b-4b4c-906e-00a6faa17df8-kube-api-access-tr7vn\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:44.014327 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:44.014231    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:44.302985 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:44.302898    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"]
Oct 30 23:28:44.306269 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:28:44.306233    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613946a6_253b_4b4c_906e_00a6faa17df8.slice/crio-8e697060fb6b9910e291f73fb2f7c7d28e6646e356a83e698ec4f10dada94c56 WatchSource:0}: Error finding container 8e697060fb6b9910e291f73fb2f7c7d28e6646e356a83e698ec4f10dada94c56: Status 404 returned error can't find the container with id 8e697060fb6b9910e291f73fb2f7c7d28e6646e356a83e698ec4f10dada94c56
Oct 30 23:28:44.518531 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:44.518256    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" event={"ID":"613946a6-253b-4b4c-906e-00a6faa17df8","Type":"ContainerStarted","Data":"a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543"}
Oct 30 23:28:44.518531 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:44.518426    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" event={"ID":"613946a6-253b-4b4c-906e-00a6faa17df8","Type":"ContainerStarted","Data":"8e697060fb6b9910e291f73fb2f7c7d28e6646e356a83e698ec4f10dada94c56"}
Oct 30 23:28:48.570419 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:48.570364    5060 generic.go:356] "Generic (PLEG): container finished" podID="613946a6-253b-4b4c-906e-00a6faa17df8" containerID="a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543" exitCode=0
Oct 30 23:28:48.577253 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:48.572573    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" event={"ID":"613946a6-253b-4b4c-906e-00a6faa17df8","Type":"ContainerDied","Data":"a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543"}
Oct 30 23:28:49.032481 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.032437    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"]
Oct 30 23:28:49.040888 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.040853    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.070533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.066402    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-fjgbt\""
Oct 30 23:28:49.070533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.066497    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Oct 30 23:28:49.070533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.066659    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Oct 30 23:28:49.070533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.066734    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Oct 30 23:28:49.070533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.067577    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:28:49.070533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.067657    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Oct 30 23:28:49.070533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.068539    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"]
Oct 30 23:28:49.159545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.141945    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-home\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.159545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.144635    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tls-certs\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.159545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.144737    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-dshm\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.159545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.144818    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.159545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.144892    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tmp-dir\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.159545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.144958    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.159545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.145141    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-model-cache\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.159545 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.145181    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw9rw\" (UniqueName: \"kubernetes.io/projected/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kube-api-access-tw9rw\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.269534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.266538    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tmp-dir\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.269534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.266698    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.269534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.266785    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-model-cache\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.269534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.267060    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw9rw\" (UniqueName: \"kubernetes.io/projected/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kube-api-access-tw9rw\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.269534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.267149    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-home\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.269534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.267183    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tls-certs\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.269534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.267208    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-dshm\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.269534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.267246    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.294067 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.289827    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.309302 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.309261    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.309812 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.309789    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tmp-dir\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.310197 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.310178    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-model-cache\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.310559 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.310539    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-home\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.340210 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.340174    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-dshm\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.340590 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.340532    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tls-certs\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.343645 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.343028    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw9rw\" (UniqueName: \"kubernetes.io/projected/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kube-api-access-tw9rw\") pod \"tls-verification-test-kserve-prefill-577b5bd485-cjmrq\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.463531 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.462931    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:49.674682 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:49.674465    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" event={"ID":"613946a6-253b-4b4c-906e-00a6faa17df8","Type":"ContainerStarted","Data":"192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355"}
Oct 30 23:28:50.015146 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:50.015045    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:50.081307 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:50.080497    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podStartSLOduration=3.61722697 podStartE2EDuration="7.080476964s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.463249994s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:49.719468145 +0000 UTC m=+4210.539205965" watchObservedRunningTime="2026-10-30 23:28:50.080476964 +0000 UTC m=+4210.900214784"
Oct 30 23:28:50.081923 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:50.081834    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"]
Oct 30 23:28:50.089220 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:28:50.089148    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6f7b92_edd2_4257_98ce_1b0f032f15e6.slice/crio-e0de0696e1e50605eb1f0a80cb5b4efca23ac7c8efa42123f26d253d70064737 WatchSource:0}: Error finding container e0de0696e1e50605eb1f0a80cb5b4efca23ac7c8efa42123f26d253d70064737: Status 404 returned error can't find the container with id e0de0696e1e50605eb1f0a80cb5b4efca23ac7c8efa42123f26d253d70064737
Oct 30 23:28:50.690184 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:50.690145    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" event={"ID":"9c6f7b92-edd2-4257-98ce-1b0f032f15e6","Type":"ContainerStarted","Data":"65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849"}
Oct 30 23:28:50.690795 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:50.690768    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" event={"ID":"9c6f7b92-edd2-4257-98ce-1b0f032f15e6","Type":"ContainerStarted","Data":"e0de0696e1e50605eb1f0a80cb5b4efca23ac7c8efa42123f26d253d70064737"}
Oct 30 23:28:53.709428 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:53.709369    5060 generic.go:356] "Generic (PLEG): container finished" podID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerID="65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849" exitCode=0
Oct 30 23:28:53.709947 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:53.709440    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" event={"ID":"9c6f7b92-edd2-4257-98ce-1b0f032f15e6","Type":"ContainerDied","Data":"65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849"}
Oct 30 23:28:54.014918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:54.014813    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:28:54.016433 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:54.016395    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:28:54.716888 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:54.716842    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" event={"ID":"9c6f7b92-edd2-4257-98ce-1b0f032f15e6","Type":"ContainerStarted","Data":"351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d"}
Oct 30 23:28:54.747237 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:54.747153    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podStartSLOduration=2.97320156 podStartE2EDuration="5.747132646s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.773931086s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:54.744213986 +0000 UTC m=+4215.563951807" watchObservedRunningTime="2026-10-30 23:28:54.747132646 +0000 UTC m=+4215.566870468"
Oct 30 23:28:59.463606 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:59.463556    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:59.464221 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:59.463624    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:28:59.465536 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:28:59.465468    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:29:04.015102 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:04.015057    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:29:04.781543 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:29:04.778571    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:29:09.464619 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:09.464568    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:29:14.015082 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:14.015034    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:29:19.464203 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:19.464156    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:29:24.015344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:24.015285    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:29:29.464163 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:29.464122    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:29:34.015153 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:34.015100    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:29:34.775592 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:29:34.775546    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:29:39.464279 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:39.464231    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:29:44.015711 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:44.015651    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:29:49.463858 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:49.463816    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:29:54.015316 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:54.015201    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:29:59.464601 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:29:59.464541    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:30:04.014934 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:04.014878    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:30:04.777079 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:30:04.777022    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:30:09.464145 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:09.464088    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:30:14.015622 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:14.015567    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:30:19.464141 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:19.464094    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:30:24.014979 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:24.014933    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:30:29.464619 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:29.464570    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:30:34.015384 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:34.015327    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:30:34.776886 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:30:34.776848    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:30:39.464552 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:39.464477    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:30:44.015522 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:44.015474    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:30:49.464092 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:49.463987    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:30:54.014734 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:54.014683    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:30:59.464365 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:30:59.464316    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:31:04.015573 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:04.015494    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:31:04.778068 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:31:04.778027    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:31:09.464303 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:09.464252    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:31:14.015253 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:14.015198    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:31:19.464534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:19.464460    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Oct 30 23:31:24.025104 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:24.025011    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:31:24.033362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:24.033067    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:31:29.473853 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:29.473815    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:31:29.481755 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:29.481719    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:31:34.775292 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:31:34.775253    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:31:40.744918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:40.744879    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"]
Oct 30 23:31:55.768326 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:55.768278    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" containerID="cri-o://192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355" gracePeriod=45
Oct 30 23:31:56.025955 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.025853    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": dial tcp 10.132.0.89:8000: connect: connection refused"
Oct 30 23:31:56.537222 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.537142    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:31:56.631988 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.631943    5060 generic.go:356] "Generic (PLEG): container finished" podID="613946a6-253b-4b4c-906e-00a6faa17df8" containerID="192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355" exitCode=0
Oct 30 23:31:56.631988 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.631985    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"
Oct 30 23:31:56.632243 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.631983    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" event={"ID":"613946a6-253b-4b4c-906e-00a6faa17df8","Type":"ContainerDied","Data":"192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355"}
Oct 30 23:31:56.632243 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.632054    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c" event={"ID":"613946a6-253b-4b4c-906e-00a6faa17df8","Type":"ContainerDied","Data":"8e697060fb6b9910e291f73fb2f7c7d28e6646e356a83e698ec4f10dada94c56"}
Oct 30 23:31:56.632243 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.632084    5060 scope.go:122] "RemoveContainer" containerID="192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355"
Oct 30 23:31:56.640802 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.640782    5060 scope.go:122] "RemoveContainer" containerID="a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543"
Oct 30 23:31:56.642946 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.642924    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-home\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-home\") pod \"613946a6-253b-4b4c-906e-00a6faa17df8\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") "
Oct 30 23:31:56.643021 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.642963    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/613946a6-253b-4b4c-906e-00a6faa17df8-tls-certs\" (UniqueName: \"kubernetes.io/secret/613946a6-253b-4b4c-906e-00a6faa17df8-tls-certs\") pod \"613946a6-253b-4b4c-906e-00a6faa17df8\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") "
Oct 30 23:31:56.643062 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.643037    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-dshm\") pod \"613946a6-253b-4b4c-906e-00a6faa17df8\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") "
Oct 30 23:31:56.643101 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.643061    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613946a6-253b-4b4c-906e-00a6faa17df8-kube-api-access-tr7vn\" (UniqueName: \"kubernetes.io/projected/613946a6-253b-4b4c-906e-00a6faa17df8-kube-api-access-tr7vn\") pod \"613946a6-253b-4b4c-906e-00a6faa17df8\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") "
Oct 30 23:31:56.643228 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.643201    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-model-cache\") pod \"613946a6-253b-4b4c-906e-00a6faa17df8\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") "
Oct 30 23:31:56.643306 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.643277    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613946a6-253b-4b4c-906e-00a6faa17df8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/613946a6-253b-4b4c-906e-00a6faa17df8-cabundle-cert\") pod \"613946a6-253b-4b4c-906e-00a6faa17df8\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") "
Oct 30 23:31:56.643360 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.643311    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-tmp-dir\") pod \"613946a6-253b-4b4c-906e-00a6faa17df8\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") "
Oct 30 23:31:56.643360 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.643341    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-kserve-provision-location\") pod \"613946a6-253b-4b4c-906e-00a6faa17df8\" (UID: \"613946a6-253b-4b4c-906e-00a6faa17df8\") "
Oct 30 23:31:56.647541 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.643613    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-home" pod "613946a6-253b-4b4c-906e-00a6faa17df8" (UID: "613946a6-253b-4b4c-906e-00a6faa17df8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:31:56.647541 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.643857    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:31:56.647541 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.643942    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-model-cache" pod "613946a6-253b-4b4c-906e-00a6faa17df8" (UID: "613946a6-253b-4b4c-906e-00a6faa17df8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:31:56.647541 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.645459    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613946a6-253b-4b4c-906e-00a6faa17df8-cabundle-cert" pod "613946a6-253b-4b4c-906e-00a6faa17df8" (UID: "613946a6-253b-4b4c-906e-00a6faa17df8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:31:56.650679 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.650620    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-dshm" pod "613946a6-253b-4b4c-906e-00a6faa17df8" (UID: "613946a6-253b-4b4c-906e-00a6faa17df8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:31:56.650679 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.650626    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613946a6-253b-4b4c-906e-00a6faa17df8-tls-certs" pod "613946a6-253b-4b4c-906e-00a6faa17df8" (UID: "613946a6-253b-4b4c-906e-00a6faa17df8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:31:56.650679 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.650659    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613946a6-253b-4b4c-906e-00a6faa17df8-kube-api-access-tr7vn" pod "613946a6-253b-4b4c-906e-00a6faa17df8" (UID: "613946a6-253b-4b4c-906e-00a6faa17df8"). InnerVolumeSpecName "kube-api-access-tr7vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:31:56.650829 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.650691    5060 scope.go:122] "RemoveContainer" containerID="192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355"
Oct 30 23:31:56.651077 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:31:56.651057    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355\": container with ID starting with 192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355 not found: ID does not exist" containerID="192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355"
Oct 30 23:31:56.651135 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.651086    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355"} err="failed to get container status \"192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355\": rpc error: code = NotFound desc = could not find container \"192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355\": container with ID starting with 192f095a7c7d83ee73e60e7a40a2f14851fae994f4c7cda19e7eeac34905b355 not found: ID does not exist"
Oct 30 23:31:56.651135 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.651105    5060 scope.go:122] "RemoveContainer" containerID="a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543"
Oct 30 23:31:56.651416 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:31:56.651397    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543\": container with ID starting with a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543 not found: ID does not exist" containerID="a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543"
Oct 30 23:31:56.651477 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.651425    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543"} err="failed to get container status \"a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543\": rpc error: code = NotFound desc = could not find container \"a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543\": container with ID starting with a1a5404476608eead545f0b6026186b4af7a05504db97e4d75f489b7ae1ef543 not found: ID does not exist"
Oct 30 23:31:56.662885 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.662844    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-tmp-dir" pod "613946a6-253b-4b4c-906e-00a6faa17df8" (UID: "613946a6-253b-4b4c-906e-00a6faa17df8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:31:56.665367 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.665340    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-kserve-provision-location" pod "613946a6-253b-4b4c-906e-00a6faa17df8" (UID: "613946a6-253b-4b4c-906e-00a6faa17df8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:31:56.744577 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.744522    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:31:56.744577 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.744567    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr7vn\" (UniqueName: \"kubernetes.io/projected/613946a6-253b-4b4c-906e-00a6faa17df8-kube-api-access-tr7vn\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:31:56.744577 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.744581    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:31:56.744905 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.744590    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/613946a6-253b-4b4c-906e-00a6faa17df8-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:31:56.744905 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.744600    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:31:56.744905 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.744608    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/613946a6-253b-4b4c-906e-00a6faa17df8-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:31:56.744905 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.744617    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613946a6-253b-4b4c-906e-00a6faa17df8-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:31:56.958462 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.958424    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"]
Oct 30 23:31:56.963216 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:56.963175    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-66684585442w75c"]
Oct 30 23:31:57.289445 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:31:57.289353    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" path="/var/lib/kubelet/pods/613946a6-253b-4b4c-906e-00a6faa17df8/volumes"
Oct 30 23:32:04.775913 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:32:04.775875    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:32:34.776045 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:32:34.776000    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:32:41.023725 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:32:41.023681    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" podUID="a5c727ac326546386befce51c4d99c6d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:33:04.775539 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:33:04.775443    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:33:34.775181 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:33:34.775137    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:33:39.843092 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:33:39.843062    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:33:39.866531 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:33:39.866483    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:33:47.347116 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:33:47.347083    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-tls-resources-e4cad0c3_tls-verification-test-kserve-prefill-577b5bd485-cjmrq_9c6f7b92-edd2-4257-98ce-1b0f032f15e6/storage-initializer/0.log"
Oct 30 23:33:47.370880 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:33:47.370841    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-tls-resources-e4cad0c3_tls-verification-test-kserve-prefill-577b5bd485-cjmrq_9c6f7b92-edd2-4257-98ce-1b0f032f15e6/main/0.log"
Oct 30 23:33:51.150903 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:33:51.150862    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"]
Oct 30 23:34:04.775837 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:34:04.775798    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:34:06.174947 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.174894    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main" containerID="cri-o://351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d" gracePeriod=45
Oct 30 23:34:06.846803 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.846770    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:34:06.981583 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.981550    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kube-api-access-tw9rw\" (UniqueName: \"kubernetes.io/projected/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kube-api-access-tw9rw\") pod \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") "
Oct 30 23:34:06.981775 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.981621    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kserve-provision-location\") pod \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") "
Oct 30 23:34:06.981775 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.981660    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-model-cache\") pod \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") "
Oct 30 23:34:06.981775 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.981683    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-home\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-home\") pod \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") "
Oct 30 23:34:06.981775 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.981700    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tmp-dir\") pod \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") "
Oct 30 23:34:06.981775 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.981742    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tls-certs\") pod \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") "
Oct 30 23:34:06.981775 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.981768    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-cabundle-cert\") pod \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") "
Oct 30 23:34:06.982080 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.981801    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-dshm\") pod \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\" (UID: \"9c6f7b92-edd2-4257-98ce-1b0f032f15e6\") "
Oct 30 23:34:06.982080 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.981954    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-model-cache" pod "9c6f7b92-edd2-4257-98ce-1b0f032f15e6" (UID: "9c6f7b92-edd2-4257-98ce-1b0f032f15e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:06.982184 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.982156    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:06.982297 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.982266    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-cabundle-cert" pod "9c6f7b92-edd2-4257-98ce-1b0f032f15e6" (UID: "9c6f7b92-edd2-4257-98ce-1b0f032f15e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:34:06.982766 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.982733    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-home" pod "9c6f7b92-edd2-4257-98ce-1b0f032f15e6" (UID: "9c6f7b92-edd2-4257-98ce-1b0f032f15e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:06.984137 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.984110    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-dshm" pod "9c6f7b92-edd2-4257-98ce-1b0f032f15e6" (UID: "9c6f7b92-edd2-4257-98ce-1b0f032f15e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:06.984243 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.984177    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tls-certs" pod "9c6f7b92-edd2-4257-98ce-1b0f032f15e6" (UID: "9c6f7b92-edd2-4257-98ce-1b0f032f15e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:34:06.984243 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.984225    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kube-api-access-tw9rw" pod "9c6f7b92-edd2-4257-98ce-1b0f032f15e6" (UID: "9c6f7b92-edd2-4257-98ce-1b0f032f15e6"). InnerVolumeSpecName "kube-api-access-tw9rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:06.994660 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:06.994624    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tmp-dir" pod "9c6f7b92-edd2-4257-98ce-1b0f032f15e6" (UID: "9c6f7b92-edd2-4257-98ce-1b0f032f15e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:07.003496 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.003458    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kserve-provision-location" pod "9c6f7b92-edd2-4257-98ce-1b0f032f15e6" (UID: "9c6f7b92-edd2-4257-98ce-1b0f032f15e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:07.082934 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.082893    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:07.082934 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.082942    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:07.083141 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.082951    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:07.083141 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.082960    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:07.083141 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.082969    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:07.083141 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.082978    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw9rw\" (UniqueName: \"kubernetes.io/projected/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kube-api-access-tw9rw\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:07.083141 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.082989    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c6f7b92-edd2-4257-98ce-1b0f032f15e6-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:07.234470 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.234369    5060 generic.go:356] "Generic (PLEG): container finished" podID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerID="351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d" exitCode=0
Oct 30 23:34:07.234470 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.234432    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"
Oct 30 23:34:07.234470 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.234454    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" event={"ID":"9c6f7b92-edd2-4257-98ce-1b0f032f15e6","Type":"ContainerDied","Data":"351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d"}
Oct 30 23:34:07.235108 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.234535    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq" event={"ID":"9c6f7b92-edd2-4257-98ce-1b0f032f15e6","Type":"ContainerDied","Data":"e0de0696e1e50605eb1f0a80cb5b4efca23ac7c8efa42123f26d253d70064737"}
Oct 30 23:34:07.235108 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.234561    5060 scope.go:122] "RemoveContainer" containerID="351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d"
Oct 30 23:34:07.243745 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.243728    5060 scope.go:122] "RemoveContainer" containerID="65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849"
Oct 30 23:34:07.251668 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.251642    5060 scope.go:122] "RemoveContainer" containerID="351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d"
Oct 30 23:34:07.251962 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:34:07.251941    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d\": container with ID starting with 351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d not found: ID does not exist" containerID="351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d"
Oct 30 23:34:07.252024 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.251972    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d"} err="failed to get container status \"351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d\": rpc error: code = NotFound desc = could not find container \"351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d\": container with ID starting with 351c18874c36d66b83ad913afb46af8eaf14bd38a029ca8c5f558728c949a49d not found: ID does not exist"
Oct 30 23:34:07.252024 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.251990    5060 scope.go:122] "RemoveContainer" containerID="65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849"
Oct 30 23:34:07.252246 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:34:07.252222    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849\": container with ID starting with 65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849 not found: ID does not exist" containerID="65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849"
Oct 30 23:34:07.252295 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.252250    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849"} err="failed to get container status \"65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849\": rpc error: code = NotFound desc = could not find container \"65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849\": container with ID starting with 65546ecf35f2ba0512294a84f7df431e5b5460e2a0df0909371b7c9a386ac849 not found: ID does not exist"
Oct 30 23:34:07.259417 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.259390    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"]
Oct 30 23:34:07.264269 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.264246    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-577b5bd485-cjmrq"]
Oct 30 23:34:07.289362 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:07.289336    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" path="/var/lib/kubelet/pods/9c6f7b92-edd2-4257-98ce-1b0f032f15e6/volumes"
Oct 30 23:34:11.464488 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:11.464435    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"]
Oct 30 23:34:11.465059 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:11.464742    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerName="istio-proxy" containerID="cri-o://e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d" gracePeriod=30
Oct 30 23:34:11.466205 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:11.466177    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"]
Oct 30 23:34:11.466369 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:11.466348    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" podUID="940928d0-a858-47c6-a002-c05a252588db" containerName="istio-proxy" containerID="cri-o://b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a" gracePeriod=30
Oct 30 23:34:16.702479 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.702449    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:34:16.735376 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.735315    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:34:16.868189 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868148    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940928d0-a858-47c6-a002-c05a252588db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940928d0-a858-47c6-a002-c05a252588db-istiod-ca-cert\") pod \"940928d0-a858-47c6-a002-c05a252588db\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") "
Oct 30 23:34:16.868189 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868195    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-socket\") pod \"940928d0-a858-47c6-a002-c05a252588db\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") "
Oct 30 23:34:16.868430 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868216    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-envoy\") pod \"db7f7a1f-6135-4eea-a257-787949cdaa82\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") "
Oct 30 23:34:16.868430 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868236    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-kube-api-access-4mpdc\" (UniqueName: \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-kube-api-access-4mpdc\") pod \"db7f7a1f-6135-4eea-a257-787949cdaa82\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") "
Oct 30 23:34:16.868430 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868258    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db7f7a1f-6135-4eea-a257-787949cdaa82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7f7a1f-6135-4eea-a257-787949cdaa82-istiod-ca-cert\") pod \"db7f7a1f-6135-4eea-a257-787949cdaa82\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") "
Oct 30 23:34:16.868430 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868290    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-socket\") pod \"db7f7a1f-6135-4eea-a257-787949cdaa82\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") "
Oct 30 23:34:16.868430 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868310    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-data\") pod \"db7f7a1f-6135-4eea-a257-787949cdaa82\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") "
Oct 30 23:34:16.868430 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868336    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-token\" (UniqueName: \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-token\") pod \"db7f7a1f-6135-4eea-a257-787949cdaa82\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") "
Oct 30 23:34:16.868430 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868363    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-credential-socket\") pod \"db7f7a1f-6135-4eea-a257-787949cdaa82\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") "
Oct 30 23:34:16.868430 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868389    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-podinfo\") pod \"db7f7a1f-6135-4eea-a257-787949cdaa82\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") "
Oct 30 23:34:16.868430 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868423    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-certs\") pod \"940928d0-a858-47c6-a002-c05a252588db\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") "
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868449    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-certs\") pod \"db7f7a1f-6135-4eea-a257-787949cdaa82\" (UID: \"db7f7a1f-6135-4eea-a257-787949cdaa82\") "
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868486    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-envoy\") pod \"940928d0-a858-47c6-a002-c05a252588db\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") "
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868581    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/940928d0-a858-47c6-a002-c05a252588db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940928d0-a858-47c6-a002-c05a252588db-istio-podinfo\") pod \"940928d0-a858-47c6-a002-c05a252588db\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") "
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868586    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-socket" pod "940928d0-a858-47c6-a002-c05a252588db" (UID: "940928d0-a858-47c6-a002-c05a252588db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868627    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-istio-token\" (UniqueName: \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-istio-token\") pod \"940928d0-a858-47c6-a002-c05a252588db\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") "
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868641    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-socket" pod "db7f7a1f-6135-4eea-a257-787949cdaa82" (UID: "db7f7a1f-6135-4eea-a257-787949cdaa82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868676    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-credential-socket\") pod \"940928d0-a858-47c6-a002-c05a252588db\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") "
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868703    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-credential-socket" pod "db7f7a1f-6135-4eea-a257-787949cdaa82" (UID: "db7f7a1f-6135-4eea-a257-787949cdaa82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868749    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-kube-api-access-nmcj8\" (UniqueName: \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-kube-api-access-nmcj8\") pod \"940928d0-a858-47c6-a002-c05a252588db\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") "
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868755    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940928d0-a858-47c6-a002-c05a252588db-istiod-ca-cert" pod "940928d0-a858-47c6-a002-c05a252588db" (UID: "940928d0-a858-47c6-a002-c05a252588db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:34:16.868893 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868806    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-data\") pod \"940928d0-a858-47c6-a002-c05a252588db\" (UID: \"940928d0-a858-47c6-a002-c05a252588db\") "
Oct 30 23:34:16.869440 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.868976    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7f7a1f-6135-4eea-a257-787949cdaa82-istiod-ca-cert" pod "db7f7a1f-6135-4eea-a257-787949cdaa82" (UID: "db7f7a1f-6135-4eea-a257-787949cdaa82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:34:16.869440 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.869117    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940928d0-a858-47c6-a002-c05a252588db-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.869440 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.869141    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.869440 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.869161    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.869440 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.869175    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.869440 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.869064    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-data" pod "db7f7a1f-6135-4eea-a257-787949cdaa82" (UID: "db7f7a1f-6135-4eea-a257-787949cdaa82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:16.869440 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.869271    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-data" pod "940928d0-a858-47c6-a002-c05a252588db" (UID: "940928d0-a858-47c6-a002-c05a252588db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:16.869440 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.869400    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-certs" pod "db7f7a1f-6135-4eea-a257-787949cdaa82" (UID: "db7f7a1f-6135-4eea-a257-787949cdaa82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:16.869854 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.869663    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-certs" pod "940928d0-a858-47c6-a002-c05a252588db" (UID: "940928d0-a858-47c6-a002-c05a252588db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:16.870469 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.870419    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-credential-socket" pod "940928d0-a858-47c6-a002-c05a252588db" (UID: "940928d0-a858-47c6-a002-c05a252588db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:16.871156 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.871120    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-kube-api-access-4mpdc" pod "db7f7a1f-6135-4eea-a257-787949cdaa82" (UID: "db7f7a1f-6135-4eea-a257-787949cdaa82"). InnerVolumeSpecName "kube-api-access-4mpdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:16.871367 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.871342    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-envoy" pod "db7f7a1f-6135-4eea-a257-787949cdaa82" (UID: "db7f7a1f-6135-4eea-a257-787949cdaa82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:16.871643 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.871613    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-envoy" pod "940928d0-a858-47c6-a002-c05a252588db" (UID: "940928d0-a858-47c6-a002-c05a252588db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:16.871643 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.871628    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-kube-api-access-nmcj8" pod "940928d0-a858-47c6-a002-c05a252588db" (UID: "940928d0-a858-47c6-a002-c05a252588db"). InnerVolumeSpecName "kube-api-access-nmcj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:16.871937 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.871906    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/940928d0-a858-47c6-a002-c05a252588db-istio-podinfo" pod "940928d0-a858-47c6-a002-c05a252588db" (UID: "940928d0-a858-47c6-a002-c05a252588db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:34:16.872026 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.871996    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-istio-token" pod "940928d0-a858-47c6-a002-c05a252588db" (UID: "940928d0-a858-47c6-a002-c05a252588db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:16.872310 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.872289    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-podinfo" pod "db7f7a1f-6135-4eea-a257-787949cdaa82" (UID: "db7f7a1f-6135-4eea-a257-787949cdaa82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:34:16.872384 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.872345    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-token" pod "db7f7a1f-6135-4eea-a257-787949cdaa82" (UID: "db7f7a1f-6135-4eea-a257-787949cdaa82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:16.969701 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969662    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969701 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969695    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940928d0-a858-47c6-a002-c05a252588db-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969701 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969705    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969716    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969725    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmcj8\" (UniqueName: \"kubernetes.io/projected/940928d0-a858-47c6-a002-c05a252588db-kube-api-access-nmcj8\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969735    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969744    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969752    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mpdc\" (UniqueName: \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-kube-api-access-4mpdc\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969760    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7f7a1f-6135-4eea-a257-787949cdaa82-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969769    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969777    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969787    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7f7a1f-6135-4eea-a257-787949cdaa82-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969795    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940928d0-a858-47c6-a002-c05a252588db-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.969935 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:16.969803    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7f7a1f-6135-4eea-a257-787949cdaa82-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:34:17.281448 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.281407    5060 generic.go:356] "Generic (PLEG): container finished" podID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerID="e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d" exitCode=0
Oct 30 23:34:17.281663 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.281460    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"
Oct 30 23:34:17.281663 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.281495    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" event={"ID":"db7f7a1f-6135-4eea-a257-787949cdaa82","Type":"ContainerDied","Data":"e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d"}
Oct 30 23:34:17.281663 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.281574    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66" event={"ID":"db7f7a1f-6135-4eea-a257-787949cdaa82","Type":"ContainerDied","Data":"8b6b547dbd021f7de32cc0967d58b7481d84837dc4086914694dfaab2d8d1ba1"}
Oct 30 23:34:17.281663 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.281604    5060 scope.go:122] "RemoveContainer" containerID="e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d"
Oct 30 23:34:17.286939 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.283168    5060 generic.go:356] "Generic (PLEG): container finished" podID="940928d0-a858-47c6-a002-c05a252588db" containerID="b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a" exitCode=0
Oct 30 23:34:17.286939 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.283249    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" event={"ID":"940928d0-a858-47c6-a002-c05a252588db","Type":"ContainerDied","Data":"b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a"}
Oct 30 23:34:17.286939 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.283290    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9" event={"ID":"940928d0-a858-47c6-a002-c05a252588db","Type":"ContainerDied","Data":"5512eee026512c6fe44c63bed7caa3d0b69e54a77ffd5d5c079abc8a308adeec"}
Oct 30 23:34:17.286939 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.283352    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"
Oct 30 23:34:17.295624 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.295605    5060 scope.go:122] "RemoveContainer" containerID="e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d"
Oct 30 23:34:17.295894 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:34:17.295874    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d\": container with ID starting with e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d not found: ID does not exist" containerID="e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d"
Oct 30 23:34:17.296002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.295905    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d"} err="failed to get container status \"e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d\": rpc error: code = NotFound desc = could not find container \"e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d\": container with ID starting with e790d018dcacb1d66d2aa8c2012736cef9c7986e080eb8d2ad183a07ad89592d not found: ID does not exist"
Oct 30 23:34:17.296002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.295938    5060 scope.go:122] "RemoveContainer" containerID="b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a"
Oct 30 23:34:17.304098 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.304080    5060 scope.go:122] "RemoveContainer" containerID="b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a"
Oct 30 23:34:17.304344 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:34:17.304328    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a\": container with ID starting with b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a not found: ID does not exist" containerID="b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a"
Oct 30 23:34:17.304385 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.304354    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a"} err="failed to get container status \"b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a\": rpc error: code = NotFound desc = could not find container \"b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a\": container with ID starting with b86bb23872e1659497dc00baa922f9ebc4631aacb616ba24921d9b278d65ee3a not found: ID does not exist"
Oct 30 23:34:17.310121 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.310100    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"]
Oct 30 23:34:17.314870 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.314849    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-kxfn9"]
Oct 30 23:34:17.326815 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.326795    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"]
Oct 30 23:34:17.331888 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:17.331868    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-qjn66"]
Oct 30 23:34:19.289482 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:19.289444    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940928d0-a858-47c6-a002-c05a252588db" path="/var/lib/kubelet/pods/940928d0-a858-47c6-a002-c05a252588db/volumes"
Oct 30 23:34:19.289923 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:19.289909    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" path="/var/lib/kubelet/pods/db7f7a1f-6135-4eea-a257-787949cdaa82/volumes"
Oct 30 23:34:34.775360 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:34:34.775279    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:34:41.024227 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:41.024176    5060 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal" podUID="a5c727ac326546386befce51c4d99c6d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 23:34:45.830642 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.830607    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"]
Oct 30 23:34:45.831133 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831109    5060 cpu_manager.go:414] "containerMap: removing container" podUID="940928d0-a858-47c6-a002-c05a252588db" containerName="istio-proxy"
Oct 30 23:34:45.831216 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831137    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940928d0-a858-47c6-a002-c05a252588db" containerName="istio-proxy"
Oct 30 23:34:45.831216 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831149    5060 cpu_manager.go:414] "containerMap: removing container" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="storage-initializer"
Oct 30 23:34:45.831216 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831157    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="storage-initializer"
Oct 30 23:34:45.831216 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831182    5060 cpu_manager.go:414] "containerMap: removing container" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main"
Oct 30 23:34:45.831216 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831189    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main"
Oct 30 23:34:45.831216 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831199    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main"
Oct 30 23:34:45.831216 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831212    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main"
Oct 30 23:34:45.831574 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831235    5060 cpu_manager.go:414] "containerMap: removing container" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerName="istio-proxy"
Oct 30 23:34:45.831574 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831243    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerName="istio-proxy"
Oct 30 23:34:45.831574 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831265    5060 cpu_manager.go:414] "containerMap: removing container" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="storage-initializer"
Oct 30 23:34:45.831574 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831272    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="storage-initializer"
Oct 30 23:34:45.831574 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831378    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="940928d0-a858-47c6-a002-c05a252588db" containerName="istio-proxy"
Oct 30 23:34:45.831574 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831392    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="613946a6-253b-4b4c-906e-00a6faa17df8" containerName="main"
Oct 30 23:34:45.831574 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831401    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6f7b92-edd2-4257-98ce-1b0f032f15e6" containerName="main"
Oct 30 23:34:45.831574 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.831411    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="db7f7a1f-6135-4eea-a257-787949cdaa82" containerName="istio-proxy"
Oct 30 23:34:45.835938 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.835916    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:45.840701 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.840678    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Oct 30 23:34:45.841683 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.841666    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Oct 30 23:34:45.841785 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.841745    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:34:45.841785 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.841743    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Oct 30 23:34:45.841785 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.841751    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Oct 30 23:34:45.841785 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.841762    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-46nkk\""
Oct 30 23:34:45.850740 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.850715    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"]
Oct 30 23:34:45.914024 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.913981    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-home\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:45.914024 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.914026    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-dshm\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:45.914267 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.914052    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25a8e830-609e-40a0-9c86-023c2c027026-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:45.914267 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.914123    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wscz7\" (UniqueName: \"kubernetes.io/projected/25a8e830-609e-40a0-9c86-023c2c027026-kube-api-access-wscz7\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:45.914267 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.914183    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-tmp-dir\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:45.914267 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.914200    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:45.914267 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.914219    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-model-cache\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:45.914267 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:45.914259    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a8e830-609e-40a0-9c86-023c2c027026-tls-certs\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.015344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.015297    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25a8e830-609e-40a0-9c86-023c2c027026-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.015344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.015344    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wscz7\" (UniqueName: \"kubernetes.io/projected/25a8e830-609e-40a0-9c86-023c2c027026-kube-api-access-wscz7\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.015648 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.015374    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-tmp-dir\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.015648 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.015469    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.015648 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.015552    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-model-cache\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.015648 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.015628    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a8e830-609e-40a0-9c86-023c2c027026-tls-certs\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.015915 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.015672    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-home\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.015915 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.015711    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-dshm\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.015915 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.015903    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-tmp-dir\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.016025 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.015919    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.016025 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.016006    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-model-cache\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.016097 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.016020    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-home\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.016176 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.016148    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25a8e830-609e-40a0-9c86-023c2c027026-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.017976 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.017956    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-dshm\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.018284 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.018264    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a8e830-609e-40a0-9c86-023c2c027026-tls-certs\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.028131 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.028107    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wscz7\" (UniqueName: \"kubernetes.io/projected/25a8e830-609e-40a0-9c86-023c2c027026-kube-api-access-wscz7\") pod \"prestop-hook-test-kserve-5f8f955bf6-58mtp\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.151239 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.151152    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:46.438086 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:46.438056    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"]
Oct 30 23:34:46.440324 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:34:46.440293    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a8e830_609e_40a0_9c86_023c2c027026.slice/crio-afe46baf2f9eb3ba26c8564e39d4ac74816d9ae8060cd1521947336c061fae22 WatchSource:0}: Error finding container afe46baf2f9eb3ba26c8564e39d4ac74816d9ae8060cd1521947336c061fae22: Status 404 returned error can't find the container with id afe46baf2f9eb3ba26c8564e39d4ac74816d9ae8060cd1521947336c061fae22
Oct 30 23:34:47.413701 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:47.413663    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" event={"ID":"25a8e830-609e-40a0-9c86-023c2c027026","Type":"ContainerStarted","Data":"69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4"}
Oct 30 23:34:47.414087 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:47.413715    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" event={"ID":"25a8e830-609e-40a0-9c86-023c2c027026","Type":"ContainerStarted","Data":"afe46baf2f9eb3ba26c8564e39d4ac74816d9ae8060cd1521947336c061fae22"}
Oct 30 23:34:49.425202 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:49.425115    5060 generic.go:356] "Generic (PLEG): container finished" podID="25a8e830-609e-40a0-9c86-023c2c027026" containerID="69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4" exitCode=0
Oct 30 23:34:49.425202 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:49.425157    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" event={"ID":"25a8e830-609e-40a0-9c86-023c2c027026","Type":"ContainerDied","Data":"69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4"}
Oct 30 23:34:50.431302 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:50.431269    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" event={"ID":"25a8e830-609e-40a0-9c86-023c2c027026","Type":"ContainerStarted","Data":"8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f"}
Oct 30 23:34:50.469533 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:50.466459    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podStartSLOduration=2.833426669 podStartE2EDuration="5.466439744s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.633013075s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:34:50.460203961 +0000 UTC m=+4571.279941778" watchObservedRunningTime="2026-10-30 23:34:50.466439744 +0000 UTC m=+4571.286177564"
Oct 30 23:34:51.153534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:51.152602    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:56.151841 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:56.151798    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:34:56.152959 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:34:56.152932    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:35:04.776310 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:35:04.776268    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:35:06.152002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:06.151952    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:35:11.594635 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.594595    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"]
Oct 30 23:35:11.598194 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.598167    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.614287 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.614250    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"]
Oct 30 23:35:11.665344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.665297    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.665534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.665361    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.665534 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.665476    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.665617 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.665547    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.665617 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.665588    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.665695 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.665613    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.665695 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.665672    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9tmx\" (UniqueName: \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-kube-api-access-f9tmx\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.665802 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.665782    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.665850 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.665838    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767240 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767191    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767441 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767266    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767441 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767314    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767441 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767363    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767441 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767392    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767441 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767424    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767746 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767452    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767746 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767477    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9tmx\" (UniqueName: \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-kube-api-access-f9tmx\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767746 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767534    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767892 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767832    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.767951 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.767901    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.768148 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.768124    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.768237 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.768215    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.768425 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.768401    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.769627 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.769598    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.769894 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.769876    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.780927 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.780900    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9tmx\" (UniqueName: \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-kube-api-access-f9tmx\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.786057 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.786033    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-zctx5\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:11.916370 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:11.916285    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:12.198641 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:12.198603    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"]
Oct 30 23:35:12.200111 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:35:12.200084    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08b26c9_0d51_4613_9e2c_e37be3f1f026.slice/crio-606bd4a26bb24911fd33997e6c78104afe451ed90d9b53dc76e4bd8209f76625 WatchSource:0}: Error finding container 606bd4a26bb24911fd33997e6c78104afe451ed90d9b53dc76e4bd8209f76625: Status 404 returned error can't find the container with id 606bd4a26bb24911fd33997e6c78104afe451ed90d9b53dc76e4bd8209f76625
Oct 30 23:35:12.202372 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:12.202329    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:35:12.202453 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:12.202422    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:35:12.202490 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:12.202465    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:35:12.536226 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:12.536129    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" event={"ID":"d08b26c9-0d51-4613-9e2c-e37be3f1f026","Type":"ContainerStarted","Data":"7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606"}
Oct 30 23:35:12.536226 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:12.536191    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" event={"ID":"d08b26c9-0d51-4613-9e2c-e37be3f1f026","Type":"ContainerStarted","Data":"606bd4a26bb24911fd33997e6c78104afe451ed90d9b53dc76e4bd8209f76625"}
Oct 30 23:35:12.562330 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:12.562260    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" podStartSLOduration=1.562241605 podStartE2EDuration="1.562241605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:35:12.560681218 +0000 UTC m=+4593.380419050" watchObservedRunningTime="2026-10-30 23:35:12.562241605 +0000 UTC m=+4593.381979424"
Oct 30 23:35:12.916490 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:12.916400    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:13.918902 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:13.918867    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-zctx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 23:35:13.919346 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:13.918928    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" podUID="d08b26c9-0d51-4613-9e2c-e37be3f1f026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 23:35:14.917022 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:14.916985    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-zctx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 23:35:14.917196 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:14.917042    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" podUID="d08b26c9-0d51-4613-9e2c-e37be3f1f026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 23:35:15.917333 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:15.917293    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-zctx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 23:35:15.917764 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:15.917365    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" podUID="d08b26c9-0d51-4613-9e2c-e37be3f1f026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 23:35:16.153397 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:16.152984    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:35:16.916917 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:16.916877    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-zctx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 23:35:16.917106 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:16.916941    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" podUID="d08b26c9-0d51-4613-9e2c-e37be3f1f026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 23:35:17.916849 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:17.916800    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-zctx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 23:35:17.917261 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:17.916873    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" podUID="d08b26c9-0d51-4613-9e2c-e37be3f1f026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 23:35:18.920997 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:18.920956    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:18.921495 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:18.921469    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:18.922345 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:18.922324    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:35:26.152632 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.152586    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:35:26.623992 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.623948    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"]
Oct 30 23:35:26.627891 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.627862    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.642774 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.642739    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"]
Oct 30 23:35:26.700141 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.700099    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9411a073-738c-449d-904e-df96841920ae-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.700360 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.700145    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.700360 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.700249    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.700360 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.700290    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q7wv\" (UniqueName: \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-kube-api-access-4q7wv\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.700360 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.700348    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.700503 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.700374    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9411a073-738c-449d-904e-df96841920ae-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.700503 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.700440    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.700503 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.700473    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.700503 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.700496    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.801572 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.801531    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9411a073-738c-449d-904e-df96841920ae-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.801761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.801581    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.801761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.801611    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.801761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.801636    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7wv\" (UniqueName: \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-kube-api-access-4q7wv\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.801761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.801684    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.801761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.801719    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9411a073-738c-449d-904e-df96841920ae-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.802029 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.801783    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.802029 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.801810    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.802029 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.801847    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.802180 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.802106    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-data\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.802439 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.802410    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-certs\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.802543 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.802447    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.802609 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.802535    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9411a073-738c-449d-904e-df96841920ae-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.802681 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.802664    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-credential-socket\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.803969 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.803941    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-envoy\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.804072 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.804014    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9411a073-738c-449d-904e-df96841920ae-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.814763 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.814734    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-istio-token\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.814904 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.814886    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7wv\" (UniqueName: \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-kube-api-access-4q7wv\") pod \"openshift-ai-inference-openshift-default-86b669b864-bv98j\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:26.945424 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:26.945392    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:27.226892 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:27.226811    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"]
Oct 30 23:35:27.230604 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:35:27.230572    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411a073_738c_449d_904e_df96841920ae.slice/crio-aac4022a54a5b24aced2cae1ea3fa7f1eb7206e6cc5120f5e59422de12caaac5 WatchSource:0}: Error finding container aac4022a54a5b24aced2cae1ea3fa7f1eb7206e6cc5120f5e59422de12caaac5: Status 404 returned error can't find the container with id aac4022a54a5b24aced2cae1ea3fa7f1eb7206e6cc5120f5e59422de12caaac5
Oct 30 23:35:27.232657 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:27.232621    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:35:27.232732 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:27.232691    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:35:27.232732 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:27.232720    5060 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:35:27.636627 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:27.636530    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" event={"ID":"9411a073-738c-449d-904e-df96841920ae","Type":"ContainerStarted","Data":"8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8"}
Oct 30 23:35:27.636627 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:27.636591    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" event={"ID":"9411a073-738c-449d-904e-df96841920ae","Type":"ContainerStarted","Data":"aac4022a54a5b24aced2cae1ea3fa7f1eb7206e6cc5120f5e59422de12caaac5"}
Oct 30 23:35:27.945928 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:27.945897    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:28.948409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:28.948374    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-bv98j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:35:28.949031 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:28.948999    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" podUID="9411a073-738c-449d-904e-df96841920ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 23:35:29.946316 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:29.946280    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-bv98j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:35:29.946614 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:29.946583    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" podUID="9411a073-738c-449d-904e-df96841920ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 23:35:30.945728 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:30.945688    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-bv98j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:35:30.946108 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:30.945743    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" podUID="9411a073-738c-449d-904e-df96841920ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 23:35:31.946039 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:31.946002    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-bv98j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:35:31.946442 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:31.946058    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" podUID="9411a073-738c-449d-904e-df96841920ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 23:35:32.946376 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:32.946340    5060 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-86b669b864-bv98j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 23:35:32.946867 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:32.946395    5060 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" podUID="9411a073-738c-449d-904e-df96841920ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 23:35:33.949426 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:33.949393    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:33.949846 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:33.949741    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:33.950551 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:33.950530    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:35:33.976050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:33.975989    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" podStartSLOduration=7.975975501 podStartE2EDuration="7.975975501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:35:27.667654497 +0000 UTC m=+4608.487392364" watchObservedRunningTime="2026-10-30 23:35:33.975975501 +0000 UTC m=+4614.795713323"
Oct 30 23:35:34.776413 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:35:34.776375    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:35:36.151698 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:36.151652    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:35:46.152188 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:46.152144    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:35:56.152252 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:35:56.152150    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:36:04.775983 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:36:04.775944    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:36:06.151862 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:06.151813    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:36:16.152678 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:16.152618    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:36:26.152193 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.152151    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:36:26.484702 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.484667    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"]
Oct 30 23:36:26.499736 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.499701    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:36:26.501279 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.501248    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"]
Oct 30 23:36:26.558715 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.558673    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 23:36:26.562562 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.562532    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.577396 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.577317    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 23:36:26.591160 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.591121    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr\" (UID: \"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:36:26.591335 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.591193    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qrw\" (UniqueName: \"kubernetes.io/projected/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-kube-api-access-b2qrw\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr\" (UID: \"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:36:26.604447 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.604405    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Oct 30 23:36:26.604752 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.604734    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 23:36:26.604822 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.604758    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Oct 30 23:36:26.604822 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.604784    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Oct 30 23:36:26.604926 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.604837    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Oct 30 23:36:26.604971 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.604930    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-5j2b2\""
Oct 30 23:36:26.605028 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.604974    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-fjgh6\""
Oct 30 23:36:26.605078 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.605039    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-rmwcw\""
Oct 30 23:36:26.692239 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.692200    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.692239 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.692243    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr\" (UID: \"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:36:26.692467 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.692268    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qrw\" (UniqueName: \"kubernetes.io/projected/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-kube-api-access-b2qrw\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr\" (UID: \"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:36:26.692467 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.692337    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.692467 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.692384    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.692467 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.692463    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.692645 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.692490    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlpg\" (UniqueName: \"kubernetes.io/projected/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kube-api-access-xvlpg\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.692645 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.692561    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.692645 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.692603    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.692645 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.692627    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.695116 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.695088    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr\" (UID: \"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:36:26.705090 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.705060    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qrw\" (UniqueName: \"kubernetes.io/projected/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-kube-api-access-b2qrw\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr\" (UID: \"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:36:26.793586 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.793457    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.793586 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.793529    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.793586 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.793556    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.793586 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.793577    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.793917 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.793594    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlpg\" (UniqueName: \"kubernetes.io/projected/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kube-api-access-xvlpg\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.793917 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.793620    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.793917 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.793646    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.793917 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.793665    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.794130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.794034    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.794130 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.794086    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.794376 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.794317    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.794376 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.794321    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.794764 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.794440    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.796172 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.796153    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.796348 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.796331    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.806785 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.806755    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlpg\" (UniqueName: \"kubernetes.io/projected/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kube-api-access-xvlpg\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:26.816713 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.816681    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:36:26.881821 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:26.881784    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:36:27.109867 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:27.109830    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"]
Oct 30 23:36:27.112883 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:36:27.112845    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3408fd7_310b_44f7_a6a1_5b5cdac3cc55.slice/crio-f2b887bd43cf3d0997f9699b7e51e16be0840fbe1ae92fc6fd4d2ff80f64bf45 WatchSource:0}: Error finding container f2b887bd43cf3d0997f9699b7e51e16be0840fbe1ae92fc6fd4d2ff80f64bf45: Status 404 returned error can't find the container with id f2b887bd43cf3d0997f9699b7e51e16be0840fbe1ae92fc6fd4d2ff80f64bf45
Oct 30 23:36:27.178155 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:27.178125    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 23:36:27.180232 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:36:27.180200    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bec7da8_2fb8_4a99_9001_7ea2292c0d07.slice/crio-d6514afbccf42568fb921a5b8ea0441718a55b8de7fcd411eafffe6684a88179 WatchSource:0}: Error finding container d6514afbccf42568fb921a5b8ea0441718a55b8de7fcd411eafffe6684a88179: Status 404 returned error can't find the container with id d6514afbccf42568fb921a5b8ea0441718a55b8de7fcd411eafffe6684a88179
Oct 30 23:36:27.964981 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:27.964926    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7bec7da8-2fb8-4a99-9001-7ea2292c0d07","Type":"ContainerStarted","Data":"1b0f6eb1b72927359b165520ab99877ba9728e23d10c0209126143e0bcd036d2"}
Oct 30 23:36:27.965196 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:27.965023    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7bec7da8-2fb8-4a99-9001-7ea2292c0d07","Type":"ContainerStarted","Data":"d6514afbccf42568fb921a5b8ea0441718a55b8de7fcd411eafffe6684a88179"}
Oct 30 23:36:27.966668 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:27.966636    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr" event={"ID":"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55","Type":"ContainerStarted","Data":"508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b"}
Oct 30 23:36:27.966821 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:27.966691    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr" event={"ID":"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55","Type":"ContainerStarted","Data":"f2b887bd43cf3d0997f9699b7e51e16be0840fbe1ae92fc6fd4d2ff80f64bf45"}
Oct 30 23:36:27.966821 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:27.966729    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:36:28.012087 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:28.012025    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr" podStartSLOduration=2.012009937 podStartE2EDuration="2.012009937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:36:28.011176194 +0000 UTC m=+4668.830914013" watchObservedRunningTime="2026-10-30 23:36:28.012009937 +0000 UTC m=+4668.831747756"
Oct 30 23:36:29.979195 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:29.979142    5060 generic.go:356] "Generic (PLEG): container finished" podID="7bec7da8-2fb8-4a99-9001-7ea2292c0d07" containerID="1b0f6eb1b72927359b165520ab99877ba9728e23d10c0209126143e0bcd036d2" exitCode=0
Oct 30 23:36:29.979581 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:29.979229    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7bec7da8-2fb8-4a99-9001-7ea2292c0d07","Type":"ContainerDied","Data":"1b0f6eb1b72927359b165520ab99877ba9728e23d10c0209126143e0bcd036d2"}
Oct 30 23:36:30.987404 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:30.987366    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7bec7da8-2fb8-4a99-9001-7ea2292c0d07","Type":"ContainerStarted","Data":"51c5d4474f6bb7f184af0eb7ef503da91f80f2e2e0d81cb740ea721de048c9ad"}
Oct 30 23:36:31.015468 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:31.015416    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.412466771 podStartE2EDuration="5.015401044s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.602934273s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:36:31.013033079 +0000 UTC m=+4671.832770912" watchObservedRunningTime="2026-10-30 23:36:31.015401044 +0000 UTC m=+4671.835138863"
Oct 30 23:36:34.776684 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:36:34.776640    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:36:36.152586 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:36.152538    5060 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:36:46.161925 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:46.161890    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:36:46.170083 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:46.170044    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:36:48.762978 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:48.762936    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"]
Oct 30 23:36:58.975685 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:36:58.975650    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:37:03.786333 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:03.786293    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" containerID="cri-o://8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f" gracePeriod=45
Oct 30 23:37:04.162656 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.162560    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Oct 30 23:37:04.541985 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.541954    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:37:04.653805 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.653768    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-tmp-dir\") pod \"25a8e830-609e-40a0-9c86-023c2c027026\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") "
Oct 30 23:37:04.653981 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.653849    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-home\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-home\") pod \"25a8e830-609e-40a0-9c86-023c2c027026\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") "
Oct 30 23:37:04.653981 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.653878    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-kserve-provision-location\") pod \"25a8e830-609e-40a0-9c86-023c2c027026\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") "
Oct 30 23:37:04.653981 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.653938    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25a8e830-609e-40a0-9c86-023c2c027026-tls-certs\" (UniqueName: \"kubernetes.io/secret/25a8e830-609e-40a0-9c86-023c2c027026-tls-certs\") pod \"25a8e830-609e-40a0-9c86-023c2c027026\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") "
Oct 30 23:37:04.654171 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.653998    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a8e830-609e-40a0-9c86-023c2c027026-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25a8e830-609e-40a0-9c86-023c2c027026-cabundle-cert\") pod \"25a8e830-609e-40a0-9c86-023c2c027026\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") "
Oct 30 23:37:04.654171 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.654029    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-model-cache\") pod \"25a8e830-609e-40a0-9c86-023c2c027026\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") "
Oct 30 23:37:04.654171 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.654069    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a8e830-609e-40a0-9c86-023c2c027026-kube-api-access-wscz7\" (UniqueName: \"kubernetes.io/projected/25a8e830-609e-40a0-9c86-023c2c027026-kube-api-access-wscz7\") pod \"25a8e830-609e-40a0-9c86-023c2c027026\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") "
Oct 30 23:37:04.654171 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.654093    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-dshm\") pod \"25a8e830-609e-40a0-9c86-023c2c027026\" (UID: \"25a8e830-609e-40a0-9c86-023c2c027026\") "
Oct 30 23:37:04.654615 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.654488    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-model-cache" pod "25a8e830-609e-40a0-9c86-023c2c027026" (UID: "25a8e830-609e-40a0-9c86-023c2c027026"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:37:04.654740 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.654638    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-home" pod "25a8e830-609e-40a0-9c86-023c2c027026" (UID: "25a8e830-609e-40a0-9c86-023c2c027026"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:37:04.654802 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.654753    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:37:04.654802 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.654772    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:37:04.654904 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.654836    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a8e830-609e-40a0-9c86-023c2c027026-cabundle-cert" pod "25a8e830-609e-40a0-9c86-023c2c027026" (UID: "25a8e830-609e-40a0-9c86-023c2c027026"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:37:04.657179 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.657137    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a8e830-609e-40a0-9c86-023c2c027026-kube-api-access-wscz7" pod "25a8e830-609e-40a0-9c86-023c2c027026" (UID: "25a8e830-609e-40a0-9c86-023c2c027026"). InnerVolumeSpecName "kube-api-access-wscz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:37:04.657464 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.657429    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25a8e830-609e-40a0-9c86-023c2c027026-tls-certs" pod "25a8e830-609e-40a0-9c86-023c2c027026" (UID: "25a8e830-609e-40a0-9c86-023c2c027026"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:37:04.657751 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.657593    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-dshm" pod "25a8e830-609e-40a0-9c86-023c2c027026" (UID: "25a8e830-609e-40a0-9c86-023c2c027026"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:37:04.669256 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.669217    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-tmp-dir" pod "25a8e830-609e-40a0-9c86-023c2c027026" (UID: "25a8e830-609e-40a0-9c86-023c2c027026"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:37:04.679256 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.679214    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-kserve-provision-location" pod "25a8e830-609e-40a0-9c86-023c2c027026" (UID: "25a8e830-609e-40a0-9c86-023c2c027026"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:37:04.755661 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.755568    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25a8e830-609e-40a0-9c86-023c2c027026-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:37:04.755661 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.755600    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25a8e830-609e-40a0-9c86-023c2c027026-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:37:04.755661 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.755613    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wscz7\" (UniqueName: \"kubernetes.io/projected/25a8e830-609e-40a0-9c86-023c2c027026-kube-api-access-wscz7\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:37:04.755661 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.755630    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:37:04.755661 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.755642    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:37:04.755661 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:04.755650    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25a8e830-609e-40a0-9c86-023c2c027026-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:37:04.777205 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:37:04.777162    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:37:05.169286 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.169194    5060 generic.go:356] "Generic (PLEG): container finished" podID="25a8e830-609e-40a0-9c86-023c2c027026" containerID="8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f" exitCode=0
Oct 30 23:37:05.169748 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.169289    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"
Oct 30 23:37:05.169748 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.169278    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" event={"ID":"25a8e830-609e-40a0-9c86-023c2c027026","Type":"ContainerDied","Data":"8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f"}
Oct 30 23:37:05.169748 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.169470    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp" event={"ID":"25a8e830-609e-40a0-9c86-023c2c027026","Type":"ContainerDied","Data":"afe46baf2f9eb3ba26c8564e39d4ac74816d9ae8060cd1521947336c061fae22"}
Oct 30 23:37:05.169748 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.169520    5060 scope.go:122] "RemoveContainer" containerID="8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f"
Oct 30 23:37:05.178557 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.178530    5060 scope.go:122] "RemoveContainer" containerID="69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4"
Oct 30 23:37:05.186877 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.186857    5060 scope.go:122] "RemoveContainer" containerID="8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f"
Oct 30 23:37:05.187170 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:37:05.187150    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f\": container with ID starting with 8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f not found: ID does not exist" containerID="8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f"
Oct 30 23:37:05.187262 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.187182    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f"} err="failed to get container status \"8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f\": rpc error: code = NotFound desc = could not find container \"8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f\": container with ID starting with 8e21605374e7b0f0bbdcacdd455da79006f9608219183010fd6bcb1be522aa8f not found: ID does not exist"
Oct 30 23:37:05.187262 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.187217    5060 scope.go:122] "RemoveContainer" containerID="69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4"
Oct 30 23:37:05.187494 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:37:05.187476    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4\": container with ID starting with 69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4 not found: ID does not exist" containerID="69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4"
Oct 30 23:37:05.187573 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.187500    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4"} err="failed to get container status \"69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4\": rpc error: code = NotFound desc = could not find container \"69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4\": container with ID starting with 69526feb47b41c9babfe6c5e346fa3222e891463f09f0a0eae409b136a2d62c4 not found: ID does not exist"
Oct 30 23:37:05.198757 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.198725    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"]
Oct 30 23:37:05.205595 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.205569    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8f955bf6-58mtp"]
Oct 30 23:37:05.291591 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:37:05.291557    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a8e830-609e-40a0-9c86-023c2c027026" path="/var/lib/kubelet/pods/25a8e830-609e-40a0-9c86-023c2c027026/volumes"
Oct 30 23:37:34.777394 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:37:34.777303    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:38:04.776123 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:38:04.776082    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:38:12.393854 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:12.393812    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"]
Oct 30 23:38:18.975567 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:18.975498    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr" podUID="a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:38:27.414522 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:27.414470    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr" podUID="a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" containerName="main" containerID="cri-o://508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b" gracePeriod=45
Oct 30 23:38:28.975083 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:28.975043    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr" podUID="a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:38:28.993628 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:28.993590    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 23:38:34.775831 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:38:34.775784    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:38:38.975036 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:38.974994    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr" podUID="a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:38:38.975437 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:38.975125    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:38:39.871258 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:39.871225    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:38:39.895345 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:39.895313    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:38:44.016075 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.015966    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="7bec7da8-2fb8-4a99-9001-7ea2292c0d07" containerName="main" containerID="cri-o://51c5d4474f6bb7f184af0eb7ef503da91f80f2e2e0d81cb740ea721de048c9ad" gracePeriod=45
Oct 30 23:38:44.654582 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.654547    5060 generic.go:356] "Generic (PLEG): container finished" podID="7bec7da8-2fb8-4a99-9001-7ea2292c0d07" containerID="51c5d4474f6bb7f184af0eb7ef503da91f80f2e2e0d81cb740ea721de048c9ad" exitCode=0
Oct 30 23:38:44.654742 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.654629    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7bec7da8-2fb8-4a99-9001-7ea2292c0d07","Type":"ContainerDied","Data":"51c5d4474f6bb7f184af0eb7ef503da91f80f2e2e0d81cb740ea721de048c9ad"}
Oct 30 23:38:44.774323 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.774288    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:38:44.797050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.796955    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-home\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-home\") pod \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") "
Oct 30 23:38:44.797050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797010    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tmp-dir\") pod \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") "
Oct 30 23:38:44.797050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797035    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-model-cache\") pod \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") "
Oct 30 23:38:44.797351 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797063    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-dshm\") pod \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") "
Oct 30 23:38:44.797351 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797120    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kube-api-access-xvlpg\" (UniqueName: \"kubernetes.io/projected/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kube-api-access-xvlpg\") pod \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") "
Oct 30 23:38:44.797351 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797177    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tls-certs\") pod \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") "
Oct 30 23:38:44.797351 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797200    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-cabundle-cert\") pod \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") "
Oct 30 23:38:44.797351 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797232    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kserve-provision-location\") pod \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\" (UID: \"7bec7da8-2fb8-4a99-9001-7ea2292c0d07\") "
Oct 30 23:38:44.797351 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797303    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-model-cache" pod "7bec7da8-2fb8-4a99-9001-7ea2292c0d07" (UID: "7bec7da8-2fb8-4a99-9001-7ea2292c0d07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:44.797756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797571    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:38:44.797872 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797844    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-home" pod "7bec7da8-2fb8-4a99-9001-7ea2292c0d07" (UID: "7bec7da8-2fb8-4a99-9001-7ea2292c0d07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:44.798020 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.797874    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-cabundle-cert" pod "7bec7da8-2fb8-4a99-9001-7ea2292c0d07" (UID: "7bec7da8-2fb8-4a99-9001-7ea2292c0d07"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:38:44.799536 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.799490    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-dshm" pod "7bec7da8-2fb8-4a99-9001-7ea2292c0d07" (UID: "7bec7da8-2fb8-4a99-9001-7ea2292c0d07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:44.799730 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.799707    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kube-api-access-xvlpg" pod "7bec7da8-2fb8-4a99-9001-7ea2292c0d07" (UID: "7bec7da8-2fb8-4a99-9001-7ea2292c0d07"). InnerVolumeSpecName "kube-api-access-xvlpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:44.799983 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.799962    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tls-certs" pod "7bec7da8-2fb8-4a99-9001-7ea2292c0d07" (UID: "7bec7da8-2fb8-4a99-9001-7ea2292c0d07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:38:44.813144 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.813104    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tmp-dir" pod "7bec7da8-2fb8-4a99-9001-7ea2292c0d07" (UID: "7bec7da8-2fb8-4a99-9001-7ea2292c0d07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:44.818753 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.818718    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kserve-provision-location" pod "7bec7da8-2fb8-4a99-9001-7ea2292c0d07" (UID: "7bec7da8-2fb8-4a99-9001-7ea2292c0d07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:44.898564 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.898496    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:38:44.898564 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.898556    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:38:44.898564 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.898567    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvlpg\" (UniqueName: \"kubernetes.io/projected/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kube-api-access-xvlpg\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:38:44.898811 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.898579    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:38:44.898811 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.898589    5060 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-cabundle-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:38:44.898811 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.898603    5060 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-kserve-provision-location\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:38:44.898811 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:44.898612    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bec7da8-2fb8-4a99-9001-7ea2292c0d07-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:38:45.661689 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:45.661593    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 23:38:45.661689 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:45.661623    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7bec7da8-2fb8-4a99-9001-7ea2292c0d07","Type":"ContainerDied","Data":"d6514afbccf42568fb921a5b8ea0441718a55b8de7fcd411eafffe6684a88179"}
Oct 30 23:38:45.662282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:45.661698    5060 scope.go:122] "RemoveContainer" containerID="51c5d4474f6bb7f184af0eb7ef503da91f80f2e2e0d81cb740ea721de048c9ad"
Oct 30 23:38:45.670221 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:45.670203    5060 scope.go:122] "RemoveContainer" containerID="1b0f6eb1b72927359b165520ab99877ba9728e23d10c0209126143e0bcd036d2"
Oct 30 23:38:45.682685 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:45.682656    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 23:38:45.687079 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:45.687055    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 23:38:47.289774 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:47.289737    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bec7da8-2fb8-4a99-9001-7ea2292c0d07" path="/var/lib/kubelet/pods/7bec7da8-2fb8-4a99-9001-7ea2292c0d07/volumes"
Oct 30 23:38:48.974998 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:48.974961    5060 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr" podUID="a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:38:57.671181 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.671154    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:38:57.713350 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.713307    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-kube-api-access-b2qrw\" (UniqueName: \"kubernetes.io/projected/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-kube-api-access-b2qrw\") pod \"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55\" (UID: \"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55\") "
Oct 30 23:38:57.713543 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.713452    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-tls-certs\" (UniqueName: \"kubernetes.io/secret/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-tls-certs\") pod \"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55\" (UID: \"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55\") "
Oct 30 23:38:57.715528 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.715483    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-tls-certs" pod "a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" (UID: "a3408fd7-310b-44f7-a6a1-5b5cdac3cc55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:38:57.715702 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.715676    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-kube-api-access-b2qrw" pod "a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" (UID: "a3408fd7-310b-44f7-a6a1-5b5cdac3cc55"). InnerVolumeSpecName "kube-api-access-b2qrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:57.721918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.721735    5060 generic.go:356] "Generic (PLEG): container finished" podID="a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" containerID="508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b" exitCode=0
Oct 30 23:38:57.721918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.721779    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"
Oct 30 23:38:57.721918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.721815    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr" event={"ID":"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55","Type":"ContainerDied","Data":"508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b"}
Oct 30 23:38:57.721918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.721874    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr" event={"ID":"a3408fd7-310b-44f7-a6a1-5b5cdac3cc55","Type":"ContainerDied","Data":"f2b887bd43cf3d0997f9699b7e51e16be0840fbe1ae92fc6fd4d2ff80f64bf45"}
Oct 30 23:38:57.721918 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.721906    5060 scope.go:122] "RemoveContainer" containerID="508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b"
Oct 30 23:38:57.735269 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.735247    5060 scope.go:122] "RemoveContainer" containerID="508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b"
Oct 30 23:38:57.735589 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:38:57.735568    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b\": container with ID starting with 508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b not found: ID does not exist" containerID="508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b"
Oct 30 23:38:57.735664 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.735603    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b"} err="failed to get container status \"508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b\": rpc error: code = NotFound desc = could not find container \"508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b\": container with ID starting with 508fb1ab2471a2bc7e19f4acf1c9351dfa3c97671586b37a085d6f2825237d5b not found: ID does not exist"
Oct 30 23:38:57.748184 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.748152    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"]
Oct 30 23:38:57.753073 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.753044    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2dsrr"]
Oct 30 23:38:57.814684 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.814643    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qrw\" (UniqueName: \"kubernetes.io/projected/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-kube-api-access-b2qrw\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:38:57.814684 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:57.814678    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:38:59.289469 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:38:59.289433    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" path="/var/lib/kubelet/pods/a3408fd7-310b-44f7-a6a1-5b5cdac3cc55/volumes"
Oct 30 23:39:04.776707 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:39:04.776667    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:39:09.272931 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.272894    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"]
Oct 30 23:39:09.273344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273272    5060 cpu_manager.go:414] "containerMap: removing container" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main"
Oct 30 23:39:09.273344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273283    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main"
Oct 30 23:39:09.273344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273304    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7bec7da8-2fb8-4a99-9001-7ea2292c0d07" containerName="storage-initializer"
Oct 30 23:39:09.273344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273309    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bec7da8-2fb8-4a99-9001-7ea2292c0d07" containerName="storage-initializer"
Oct 30 23:39:09.273344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273321    5060 cpu_manager.go:414] "containerMap: removing container" podUID="7bec7da8-2fb8-4a99-9001-7ea2292c0d07" containerName="main"
Oct 30 23:39:09.273344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273326    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bec7da8-2fb8-4a99-9001-7ea2292c0d07" containerName="main"
Oct 30 23:39:09.273344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273338    5060 cpu_manager.go:414] "containerMap: removing container" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="storage-initializer"
Oct 30 23:39:09.273344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273344    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="storage-initializer"
Oct 30 23:39:09.273344 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273350    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" containerName="main"
Oct 30 23:39:09.273665 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273354    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" containerName="main"
Oct 30 23:39:09.273665 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273407    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3408fd7-310b-44f7-a6a1-5b5cdac3cc55" containerName="main"
Oct 30 23:39:09.273665 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273417    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a8e830-609e-40a0-9c86-023c2c027026" containerName="main"
Oct 30 23:39:09.273665 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.273426    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bec7da8-2fb8-4a99-9001-7ea2292c0d07" containerName="main"
Oct 30 23:39:09.277778 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.277754    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.282405 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.282380    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Oct 30 23:39:09.283532 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.283493    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Oct 30 23:39:09.284655 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.284618    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Oct 30 23:39:09.284655 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.284625    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-x4nkf\""
Oct 30 23:39:09.301842 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.301806    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"]
Oct 30 23:39:09.424083 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.424039    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.424083 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.424088    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.424347 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.424188    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.424347 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.424305    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.424347 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.424336    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjwb\" (UniqueName: \"kubernetes.io/projected/566a4b73-cc7c-47df-9b5b-e17ead548c9b-kube-api-access-lpjwb\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.424479 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.424380    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.525282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.525189    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.525282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.525241    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjwb\" (UniqueName: \"kubernetes.io/projected/566a4b73-cc7c-47df-9b5b-e17ead548c9b-kube-api-access-lpjwb\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.525282 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.525267    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.525603 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.525330    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.525603 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.525353    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.525603 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.525404    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.526385 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.525603    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.526385 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.525787    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.526385 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.525838    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.527818 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.527796    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.528232 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.528215    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.542272 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.542236    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjwb\" (UniqueName: \"kubernetes.io/projected/566a4b73-cc7c-47df-9b5b-e17ead548c9b-kube-api-access-lpjwb\") pod \"rolling-upgrade-test-kserve-5d77dc844-p7l2s\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.591908 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.591866    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:09.870207 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:09.870181    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"]
Oct 30 23:39:09.871567 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:39:09.871537    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566a4b73_cc7c_47df_9b5b_e17ead548c9b.slice/crio-f3b2e82a0570ec487537b601ae879ae089077ffb38ecf07bd1ef674a365e7b5f WatchSource:0}: Error finding container f3b2e82a0570ec487537b601ae879ae089077ffb38ecf07bd1ef674a365e7b5f: Status 404 returned error can't find the container with id f3b2e82a0570ec487537b601ae879ae089077ffb38ecf07bd1ef674a365e7b5f
Oct 30 23:39:10.820991 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:10.820938    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s" event={"ID":"566a4b73-cc7c-47df-9b5b-e17ead548c9b","Type":"ContainerStarted","Data":"9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a"}
Oct 30 23:39:10.821670 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:10.821645    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s" event={"ID":"566a4b73-cc7c-47df-9b5b-e17ead548c9b","Type":"ContainerStarted","Data":"f3b2e82a0570ec487537b601ae879ae089077ffb38ecf07bd1ef674a365e7b5f"}
Oct 30 23:39:10.858686 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:10.858617    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s" podStartSLOduration=1.8585978600000002 podStartE2EDuration="1.85859786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:10.849276035 +0000 UTC m=+4831.669013856" watchObservedRunningTime="2026-10-30 23:39:10.85859786 +0000 UTC m=+4831.678335682"
Oct 30 23:39:11.595654 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:11.595610    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:19.593054 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:19.593009    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:19.605440 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:19.605409    5060 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:19.873177 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:19.873094    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:34.776294 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:39:34.776255    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:39:50.304778 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.304729    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"]
Oct 30 23:39:50.314304 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:39:50.314269    5060 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a" cmd=["/bin/sleep","15"]
Oct 30 23:39:50.314523 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.314321    5060 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 23:39:50.314523 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:39:50.314334    5060 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s" podUID="566a4b73-cc7c-47df-9b5b-e17ead548c9b" containerName="main" containerID="cri-o://9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a"
Oct 30 23:39:50.314523 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.314371    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s" podUID="566a4b73-cc7c-47df-9b5b-e17ead548c9b" containerName="main" containerID="cri-o://9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a" gracePeriod=60
Oct 30 23:39:50.674899 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.674865    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:50.742780 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.742745    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-model-cache\") pod \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") "
Oct 30 23:39:50.742972 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.742830    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tls-certs\" (UniqueName: \"kubernetes.io/secret/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tls-certs\") pod \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") "
Oct 30 23:39:50.742972 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.742886    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-dshm\") pod \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") "
Oct 30 23:39:50.743073 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.743004    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-model-cache" pod "566a4b73-cc7c-47df-9b5b-e17ead548c9b" (UID: "566a4b73-cc7c-47df-9b5b-e17ead548c9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:39:50.743073 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.743059    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-home\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-home\") pod \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") "
Oct 30 23:39:50.743178 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.743099    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tmp-dir\") pod \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") "
Oct 30 23:39:50.743244 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.743178    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566a4b73-cc7c-47df-9b5b-e17ead548c9b-kube-api-access-lpjwb\" (UniqueName: \"kubernetes.io/projected/566a4b73-cc7c-47df-9b5b-e17ead548c9b-kube-api-access-lpjwb\") pod \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\" (UID: \"566a4b73-cc7c-47df-9b5b-e17ead548c9b\") "
Oct 30 23:39:50.743529 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.743463    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tmp-dir" pod "566a4b73-cc7c-47df-9b5b-e17ead548c9b" (UID: "566a4b73-cc7c-47df-9b5b-e17ead548c9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:39:50.743529 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.743491    5060 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-model-cache\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:50.743964 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.743937    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-home" pod "566a4b73-cc7c-47df-9b5b-e17ead548c9b" (UID: "566a4b73-cc7c-47df-9b5b-e17ead548c9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:39:50.745728 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.745690    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-dshm" pod "566a4b73-cc7c-47df-9b5b-e17ead548c9b" (UID: "566a4b73-cc7c-47df-9b5b-e17ead548c9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:39:50.745821 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.745736    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tls-certs" pod "566a4b73-cc7c-47df-9b5b-e17ead548c9b" (UID: "566a4b73-cc7c-47df-9b5b-e17ead548c9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:39:50.745879 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.745807    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566a4b73-cc7c-47df-9b5b-e17ead548c9b-kube-api-access-lpjwb" pod "566a4b73-cc7c-47df-9b5b-e17ead548c9b" (UID: "566a4b73-cc7c-47df-9b5b-e17ead548c9b"). InnerVolumeSpecName "kube-api-access-lpjwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:39:50.844030 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.843940    5060 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tls-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:50.844030 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.843973    5060 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-dshm\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:50.844030 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.843982    5060 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-home\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:50.844030 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.844009    5060 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566a4b73-cc7c-47df-9b5b-e17ead548c9b-tmp-dir\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:50.844030 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:50.844018    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpjwb\" (UniqueName: \"kubernetes.io/projected/566a4b73-cc7c-47df-9b5b-e17ead548c9b-kube-api-access-lpjwb\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:39:51.001551 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:51.001480    5060 generic.go:356] "Generic (PLEG): container finished" podID="566a4b73-cc7c-47df-9b5b-e17ead548c9b" containerID="9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a" exitCode=0
Oct 30 23:39:51.001731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:51.001561    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"
Oct 30 23:39:51.001731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:51.001569    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s" event={"ID":"566a4b73-cc7c-47df-9b5b-e17ead548c9b","Type":"ContainerDied","Data":"9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a"}
Oct 30 23:39:51.001731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:51.001627    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s" event={"ID":"566a4b73-cc7c-47df-9b5b-e17ead548c9b","Type":"ContainerDied","Data":"f3b2e82a0570ec487537b601ae879ae089077ffb38ecf07bd1ef674a365e7b5f"}
Oct 30 23:39:51.001731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:51.001650    5060 scope.go:122] "RemoveContainer" containerID="9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a"
Oct 30 23:39:51.018267 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:51.018239    5060 scope.go:122] "RemoveContainer" containerID="9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a"
Oct 30 23:39:51.018910 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:39:51.018880    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a\": container with ID starting with 9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a not found: ID does not exist" containerID="9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a"
Oct 30 23:39:51.019011 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:51.018920    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a"} err="failed to get container status \"9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a\": rpc error: code = NotFound desc = could not find container \"9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a\": container with ID starting with 9a1c6a61da9f927f0b1fd30709fef9cad81794dd5ad17a5fba053c2f2923631a not found: ID does not exist"
Oct 30 23:39:51.039580 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:51.039545    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"]
Oct 30 23:39:51.047955 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:51.047924    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-p7l2s"]
Oct 30 23:39:51.289154 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:39:51.289117    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566a4b73-cc7c-47df-9b5b-e17ead548c9b" path="/var/lib/kubelet/pods/566a4b73-cc7c-47df-9b5b-e17ead548c9b/volumes"
Oct 30 23:40:04.775093 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:40:04.775054    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:40:34.775315 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:40:34.775232    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:40:55.007933 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.007890    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lzrz9/must-gather-x675w"]
Oct 30 23:40:55.008405 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.008252    5060 cpu_manager.go:414] "containerMap: removing container" podUID="566a4b73-cc7c-47df-9b5b-e17ead548c9b" containerName="main"
Oct 30 23:40:55.008405 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.008262    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566a4b73-cc7c-47df-9b5b-e17ead548c9b" containerName="main"
Oct 30 23:40:55.008405 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.008340    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="566a4b73-cc7c-47df-9b5b-e17ead548c9b" containerName="main"
Oct 30 23:40:55.011584 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.011565    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:40:55.022297 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.022259    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lzrz9\"/\"kube-root-ca.crt\""
Oct 30 23:40:55.022297 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.022261    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lzrz9\"/\"default-dockercfg-gbznp\""
Oct 30 23:40:55.022297 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.022272    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lzrz9\"/\"openshift-service-ca.crt\""
Oct 30 23:40:55.023880 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.023856    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lzrz9/must-gather-x675w"]
Oct 30 23:40:55.118788 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.118739    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hjr5\" (UniqueName: \"kubernetes.io/projected/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-kube-api-access-8hjr5\") pod \"must-gather-x675w\" (UID: \"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd\") " pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:40:55.118788 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.118792    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-must-gather-output\") pod \"must-gather-x675w\" (UID: \"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd\") " pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:40:55.220050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.220006    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hjr5\" (UniqueName: \"kubernetes.io/projected/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-kube-api-access-8hjr5\") pod \"must-gather-x675w\" (UID: \"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd\") " pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:40:55.220050 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.220054    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-must-gather-output\") pod \"must-gather-x675w\" (UID: \"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd\") " pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:40:55.220428 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.220406    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-must-gather-output\") pod \"must-gather-x675w\" (UID: \"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd\") " pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:40:55.236391 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.236355    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hjr5\" (UniqueName: \"kubernetes.io/projected/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-kube-api-access-8hjr5\") pod \"must-gather-x675w\" (UID: \"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd\") " pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:40:55.325908 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.325825    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:40:55.599840 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.599808    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lzrz9/must-gather-x675w"]
Oct 30 23:40:55.602563 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:40:55.602528    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda448f0f2_e8f1_4dc4_81ef_93fb170fbbfd.slice/crio-4d19b71e933e6163a288ca682d7527ba0c02338b910f250e6029a98c471464ae WatchSource:0}: Error finding container 4d19b71e933e6163a288ca682d7527ba0c02338b910f250e6029a98c471464ae: Status 404 returned error can't find the container with id 4d19b71e933e6163a288ca682d7527ba0c02338b910f250e6029a98c471464ae
Oct 30 23:40:55.604255 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:55.604238    5060 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:40:56.316023 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:40:56.315973    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lzrz9/must-gather-x675w" event={"ID":"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd","Type":"ContainerStarted","Data":"4d19b71e933e6163a288ca682d7527ba0c02338b910f250e6029a98c471464ae"}
Oct 30 23:41:02.350974 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:41:02.350929    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lzrz9/must-gather-x675w" event={"ID":"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd","Type":"ContainerStarted","Data":"bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875"}
Oct 30 23:41:03.357947 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:41:03.357907    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lzrz9/must-gather-x675w" event={"ID":"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd","Type":"ContainerStarted","Data":"f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de"}
Oct 30 23:41:03.378815 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:41:03.378761    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lzrz9/must-gather-x675w" podStartSLOduration=2.865400293 podStartE2EDuration="9.378747615s" totalImagesPullingTime="6.513347322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:41:03.378075977 +0000 UTC m=+4944.197813795" watchObservedRunningTime="2026-10-30 23:41:03.378747615 +0000 UTC m=+4944.198485432"
Oct 30 23:41:04.776322 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:41:04.776279    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:41:34.775979 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:41:34.775942    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:42:04.775742 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:42:04.775656    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:42:34.776400 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:42:34.776353    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:43:04.775795 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:43:04.775753    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:43:10.621701 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:10.621668    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-bv98j_9411a073-738c-449d-904e-df96841920ae/istio-proxy/0.log"
Oct 30 23:43:10.711400 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:10.711367    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-rd99c_b79eadfc-8ca8-408a-9797-30a9d24dbdfa/istio-proxy/0.log"
Oct 30 23:43:10.713739 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:10.713712    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-rd99c_b79eadfc-8ca8-408a-9797-30a9d24dbdfa/istio-proxy/1.log"
Oct 30 23:43:10.773925 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:10.773889    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-zctx5_d08b26c9-0d51-4613-9e2c-e37be3f1f026/istio-proxy/0.log"
Oct 30 23:43:10.795494 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:10.795457    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6668ff59b7-vglrn_99252037-fc1d-4055-b66b-2f0e8ab910b8/router/0.log"
Oct 30 23:43:12.057717 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:12.057685    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-bv98j_9411a073-738c-449d-904e-df96841920ae/istio-proxy/0.log"
Oct 30 23:43:12.120455 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:12.120419    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-rd99c_b79eadfc-8ca8-408a-9797-30a9d24dbdfa/istio-proxy/0.log"
Oct 30 23:43:12.122116 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:12.122095    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-rd99c_b79eadfc-8ca8-408a-9797-30a9d24dbdfa/istio-proxy/1.log"
Oct 30 23:43:12.174969 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:12.174938    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-zctx5_d08b26c9-0d51-4613-9e2c-e37be3f1f026/istio-proxy/0.log"
Oct 30 23:43:12.202631 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:12.202593    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6668ff59b7-vglrn_99252037-fc1d-4055-b66b-2f0e8ab910b8/router/0.log"
Oct 30 23:43:13.172364 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:13.172332    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-68df894cb4-nk85t_43b7b42f-037e-43b2-96a9-424c67d04d69/manager/0.log"
Oct 30 23:43:13.189797 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:13.189771    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pk9xx_e7668c50-a921-4e34-8ea3-9b84bb52b435/kuadrant-console-plugin/0.log"
Oct 30 23:43:13.291956 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:13.291927    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-658c459c99-2n266_bd14ba8d-e224-4d7d-827a-0f85607e435b/manager/0.log"
Oct 30 23:43:13.965471 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:13.965436    5060 generic.go:356] "Generic (PLEG): container finished" podID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" containerID="bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875" exitCode=0
Oct 30 23:43:13.965705 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:13.965520    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lzrz9/must-gather-x675w" event={"ID":"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd","Type":"ContainerDied","Data":"bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875"}
Oct 30 23:43:13.965889 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:13.965872    5060 scope.go:122] "RemoveContainer" containerID="bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875"
Oct 30 23:43:14.981652 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:14.981621    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lzrz9_must-gather-x675w_a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd/gather/0.log"
Oct 30 23:43:19.185943 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:19.185907    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xkt2l_adb5dbb9-f122-43fc-9ef9-867e6b2bb700/global-pull-secret-syncer/0.log"
Oct 30 23:43:19.219520 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:19.219489    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8cbpq_2e346445-18a9-494c-9b3f-6c0860f631f7/connection-checker/0.log"
Oct 30 23:43:19.250289 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:19.250261    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-b4mzj_baf3e5ab-01d9-4caa-a239-b6df965e21e4/connection-checker/0.log"
Oct 30 23:43:19.281963 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:19.281940    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tsv8h_1889bdc0-508e-4f97-80cb-4a5aa6e4f67f/connection-checker/0.log"
Oct 30 23:43:19.473494 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:19.473462    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wb9bv_60457f9c-65b0-4547-9012-2fc4efe2b9e8/konnectivity-agent/0.log"
Oct 30 23:43:19.572327 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:19.572292    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-251.us-east-2.compute.internal_a5c727ac326546386befce51c4d99c6d/haproxy/0.log"
Oct 30 23:43:20.655991 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:20.655904    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-lzrz9/must-gather-x675w"]
Oct 30 23:43:20.656435 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:20.656085    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-must-gather-lzrz9/must-gather-x675w" podUID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" containerName="copy" containerID="cri-o://f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de" gracePeriod=2
Oct 30 23:43:20.660974 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:20.660943    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-lzrz9/must-gather-x675w"]
Oct 30 23:43:20.910238 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:20.910172    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lzrz9_must-gather-x675w_a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd/copy/0.log"
Oct 30 23:43:20.912761 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:20.912741    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:43:20.915320 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:43:20.915293    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-x675w\" is forbidden: User \"system:node:ip-10-0-138-251.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lzrz9\": no relationship found between node 'ip-10-0-138-251.us-east-2.compute.internal' and this object" podUID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:43:21.004434 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.004406    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lzrz9_must-gather-x675w_a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd/copy/0.log"
Oct 30 23:43:21.004753 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.004733    5060 generic.go:356] "Generic (PLEG): container finished" podID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" containerID="f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de" exitCode=143
Oct 30 23:43:21.004804 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.004782    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:43:21.004838 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.004824    5060 scope.go:122] "RemoveContainer" containerID="f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de"
Oct 30 23:43:21.007440 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:43:21.007408    5060 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-x675w\" is forbidden: User \"system:node:ip-10-0-138-251.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lzrz9\": no relationship found between node 'ip-10-0-138-251.us-east-2.compute.internal' and this object" podUID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" pod="openshift-must-gather-lzrz9/must-gather-x675w"
Oct 30 23:43:21.013151 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.013127    5060 scope.go:122] "RemoveContainer" containerID="bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875"
Oct 30 23:43:21.026684 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.026662    5060 scope.go:122] "RemoveContainer" containerID="f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de"
Oct 30 23:43:21.026949 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:43:21.026923    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de\": container with ID starting with f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de not found: ID does not exist" containerID="f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de"
Oct 30 23:43:21.027006 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.026964    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de"} err="failed to get container status \"f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de\": rpc error: code = NotFound desc = could not find container \"f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de\": container with ID starting with f6d6f4b008a1e5f27f5e0735fbc802cedf60e7e70c654add34c9143564d021de not found: ID does not exist"
Oct 30 23:43:21.027006 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.026985    5060 scope.go:122] "RemoveContainer" containerID="bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875"
Oct 30 23:43:21.027239 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:43:21.027222    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875\": container with ID starting with bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875 not found: ID does not exist" containerID="bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875"
Oct 30 23:43:21.027304 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.027245    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875"} err="failed to get container status \"bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875\": rpc error: code = NotFound desc = could not find container \"bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875\": container with ID starting with bb5b4b86a87e49bb302832e704172fe69b4a409cb67762a82afc39e541de9875 not found: ID does not exist"
Oct 30 23:43:21.062768 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.062730    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-must-gather-output\") pod \"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd\" (UID: \"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd\") "
Oct 30 23:43:21.062944 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.062834    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-kube-api-access-8hjr5\" (UniqueName: \"kubernetes.io/projected/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-kube-api-access-8hjr5\") pod \"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd\" (UID: \"a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd\") "
Oct 30 23:43:21.065070 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.065048    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-kube-api-access-8hjr5" pod "a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" (UID: "a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd"). InnerVolumeSpecName "kube-api-access-8hjr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:21.069977 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.069954    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-must-gather-output" pod "a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" (UID: "a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:21.164229 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.164137    5060 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-must-gather-output\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:21.164229 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.164171    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hjr5\" (UniqueName: \"kubernetes.io/projected/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd-kube-api-access-8hjr5\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:43:21.289840 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:21.289805    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" path="/var/lib/kubelet/pods/a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd/volumes"
Oct 30 23:43:34.775403 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:43:34.775355    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:43:39.898774 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:39.898742    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:43:39.923064 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:39.923034    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:43:52.530475 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:52.530442    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-68df894cb4-nk85t_43b7b42f-037e-43b2-96a9-424c67d04d69/manager/0.log"
Oct 30 23:43:52.564358 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:52.564327    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pk9xx_e7668c50-a921-4e34-8ea3-9b84bb52b435/kuadrant-console-plugin/0.log"
Oct 30 23:43:52.687522 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:52.687483    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-658c459c99-2n266_bd14ba8d-e224-4d7d-827a-0f85607e435b/manager/0.log"
Oct 30 23:43:53.790927 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:53.790894    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-4vs7k_c69ae355-a045-48d1-b7c9-881d2c4e8a1b/cluster-monitoring-operator/0.log"
Oct 30 23:43:54.132795 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:54.132714    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5h8g5_1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3/node-exporter/0.log"
Oct 30 23:43:54.167272 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:54.167228    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5h8g5_1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3/kube-rbac-proxy/0.log"
Oct 30 23:43:54.204882 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:54.204854    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5h8g5_1b8cfadb-aa0f-4027-92cf-00d4b1a62ed3/init-textfile/0.log"
Oct 30 23:43:54.922564 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:54.922489    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-wwjmw_18c9ba18-01c6-46e7-8e3c-de119ba81cc4/prometheus-operator-admission-webhook/0.log"
Oct 30 23:43:55.653765 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.653728    5060 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"]
Oct 30 23:43:55.654090 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.654078    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" containerName="gather"
Oct 30 23:43:55.654137 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.654092    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" containerName="gather"
Oct 30 23:43:55.654137 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.654113    5060 cpu_manager.go:414] "containerMap: removing container" podUID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" containerName="copy"
Oct 30 23:43:55.654137 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.654118    5060 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" containerName="copy"
Oct 30 23:43:55.654233 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.654180    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" containerName="gather"
Oct 30 23:43:55.654233 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.654189    5060 memory_manager.go:365] "RemoveStaleState removing state" podUID="a448f0f2-e8f1-4dc4-81ef-93fb170fbbfd" containerName="copy"
Oct 30 23:43:55.657198 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.657175    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.672414 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.672382    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"]
Oct 30 23:43:55.745518 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.745477    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-sys\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.745688 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.745625    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-proc\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.745731 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.745707    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-lib-modules\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.745766 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.745737    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-podres\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.745804 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.745770    5060 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5tw2\" (UniqueName: \"kubernetes.io/projected/9e2a26ef-7962-456f-a30d-65ed7546202f-kube-api-access-b5tw2\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.752605 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.752585    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dn4ld\"/\"kube-root-ca.crt\""
Oct 30 23:43:55.752678 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.752597    5060 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dn4ld\"/\"openshift-service-ca.crt\""
Oct 30 23:43:55.752933 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.752920    5060 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dn4ld\"/\"default-dockercfg-57457\""
Oct 30 23:43:55.846390 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.846349    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-proc\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.846616 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.846398    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-lib-modules\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.846616 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.846441    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-podres\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.846616 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.846481    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-proc\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.846616 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.846481    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5tw2\" (UniqueName: \"kubernetes.io/projected/9e2a26ef-7962-456f-a30d-65ed7546202f-kube-api-access-b5tw2\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.846616 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.846584    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-podres\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.846616 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.846532    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-lib-modules\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.846616 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.846601    5060 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-sys\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.846841 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.846690    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e2a26ef-7962-456f-a30d-65ed7546202f-sys\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.860677 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.860643    5060 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5tw2\" (UniqueName: \"kubernetes.io/projected/9e2a26ef-7962-456f-a30d-65ed7546202f-kube-api-access-b5tw2\") pod \"perf-node-gather-daemonset-6p7vb\" (UID: \"9e2a26ef-7962-456f-a30d-65ed7546202f\") " pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:55.970431 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:55.970397    5060 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:56.245144 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:56.245118    5060 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"]
Oct 30 23:43:56.247887 ip-10-0-138-251 kubenswrapper[5060]: W1030 23:43:56.247853    5060 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9e2a26ef_7962_456f_a30d_65ed7546202f.slice/crio-af07d2de1a9f86824b1d49e803be048e0344a3f022bb791079153e1a903a3c81 WatchSource:0}: Error finding container af07d2de1a9f86824b1d49e803be048e0344a3f022bb791079153e1a903a3c81: Status 404 returned error can't find the container with id af07d2de1a9f86824b1d49e803be048e0344a3f022bb791079153e1a903a3c81
Oct 30 23:43:56.418076 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:56.418041    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-grpzl_88d59947-036f-412d-9ee6-f7565b0d4d8b/networking-console-plugin/0.log"
Oct 30 23:43:56.943574 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:56.943545    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/2.log"
Oct 30 23:43:56.947396 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:56.947369    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-dd7n4_88c3c753-4382-452d-898d-732766a65bf1/console-operator/3.log"
Oct 30 23:43:57.167619 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:57.167577    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb" event={"ID":"9e2a26ef-7962-456f-a30d-65ed7546202f","Type":"ContainerStarted","Data":"cb512c7aa1d83dd409f5179d54c5b7725757888c46c06e84ac256e9c45e62679"}
Oct 30 23:43:57.168045 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:57.167642    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb" event={"ID":"9e2a26ef-7962-456f-a30d-65ed7546202f","Type":"ContainerStarted","Data":"af07d2de1a9f86824b1d49e803be048e0344a3f022bb791079153e1a903a3c81"}
Oct 30 23:43:57.168045 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:57.167668    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:43:57.200079 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:57.199977    5060 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb" podStartSLOduration=2.199963315 podStartE2EDuration="2.199963315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:57.198561395 +0000 UTC m=+5118.018299210" watchObservedRunningTime="2026-10-30 23:43:57.199963315 +0000 UTC m=+5118.019701134"
Oct 30 23:43:57.459325 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:57.459233    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-7r6gx_bd473000-e4aa-480c-bbf2-a15cfba05c40/download-server/0.log"
Oct 30 23:43:58.030837 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:58.030803    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-tw8c8_3a7d925c-af25-4c40-a280-e870b59cb772/volume-data-source-validator/0.log"
Oct 30 23:43:59.115542 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:59.115497    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6wrrt_8e71c29d-cd72-42ae-a0de-1500b78c89b0/dns/0.log"
Oct 30 23:43:59.160985 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:59.160956    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6wrrt_8e71c29d-cd72-42ae-a0de-1500b78c89b0/kube-rbac-proxy/0.log"
Oct 30 23:43:59.350988 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:59.350958    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qqwss_92a4974a-3e6e-4575-b5bd-303beb3fa201/dns-node-resolver/0.log"
Oct 30 23:43:59.967269 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:43:59.967229    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c47bcc548-5vsbs_af0582ec-cfd0-41a7-be62-ad6b912e92f3/registry/0.log"
Oct 30 23:44:00.015829 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:00.015798    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7cjdl_a3bd8549-385d-4ad6-8fac-cd2e7b4b37e6/node-ca/0.log"
Oct 30 23:44:01.519557 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:01.519501    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-bv98j_9411a073-738c-449d-904e-df96841920ae/istio-proxy/0.log"
Oct 30 23:44:01.640916 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:01.640877    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-rd99c_b79eadfc-8ca8-408a-9797-30a9d24dbdfa/istio-proxy/0.log"
Oct 30 23:44:01.642456 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:01.642432    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-rd99c_b79eadfc-8ca8-408a-9797-30a9d24dbdfa/istio-proxy/1.log"
Oct 30 23:44:01.752899 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:01.752853    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-86b669b864-zctx5_d08b26c9-0d51-4613-9e2c-e37be3f1f026/istio-proxy/0.log"
Oct 30 23:44:01.797841 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:01.797762    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6668ff59b7-vglrn_99252037-fc1d-4055-b66b-2f0e8ab910b8/router/0.log"
Oct 30 23:44:02.617733 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:02.617696    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rvz8k_188a085b-a512-440b-90ec-30acc66e4171/serve-healthcheck-canary/0.log"
Oct 30 23:44:03.125805 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:03.125770    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-xkcch_16ece3f9-76fd-4c93-b915-d1c8cec664fb/insights-operator/0.log"
Oct 30 23:44:03.127138 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:03.127117    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-xkcch_16ece3f9-76fd-4c93-b915-d1c8cec664fb/insights-operator/1.log"
Oct 30 23:44:03.183192 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:03.183164    5060 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dn4ld/perf-node-gather-daemonset-6p7vb"
Oct 30 23:44:03.253461 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:03.253434    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mwl7_d3d84917-1245-4b8c-a887-fd2d668bcd2d/kube-rbac-proxy/0.log"
Oct 30 23:44:03.286771 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:03.286736    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mwl7_d3d84917-1245-4b8c-a887-fd2d668bcd2d/exporter/0.log"
Oct 30 23:44:03.319450 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:03.319424    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6mwl7_d3d84917-1245-4b8c-a887-fd2d668bcd2d/extractor/0.log"
Oct 30 23:44:03.481199 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:03.481166    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-mxxhw-rrcns_6b5b9230-9003-4591-8440-01f8a2ccf26e/insights-gathering/0.log"
Oct 30 23:44:04.775699 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:44:04.775660    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:44:06.603755 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:06.603725    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6667976c75-tdwkq_1f872351-c46d-43a9-ba57-f7e80a2403f6/manager/0.log"
Oct 30 23:44:07.724341 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:07.724307    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-56ddc4c448-mzrn8_a4b18042-569f-4065-b5fe-17396e93712b/server/0.log"
Oct 30 23:44:08.292542 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:08.292482    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zx9kx_cc0ceeca-1486-49b2-ac12-0f17cf399052/s3-init/0.log"
Oct 30 23:44:08.329211 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:08.329176    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-zx9kx_cc0ceeca-1486-49b2-ac12-0f17cf399052/download-hf-model/0.log"
Oct 30 23:44:08.378096 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:08.378063    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-78bb9c6ff8-75jl9_90651480-9980-4318-a935-3cde92a8ef96/seaweedfs/0.log"
Oct 30 23:44:13.580234 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:13.580207    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ds7bd_16e7c643-41ee-4ac5-b285-1351ae323a1d/migrator/0.log"
Oct 30 23:44:13.615866 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:13.615833    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-vfc6j_c097d835-0be1-4736-8dd8-98f375f02f47/migrator/0.log"
Oct 30 23:44:14.098497 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:14.098464    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b_f8199465-5b50-41fd-ab4f-5201977736a3/kube-storage-version-migrator-operator/1.log"
Oct 30 23:44:14.099590 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:14.099569    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-bjp2b_f8199465-5b50-41fd-ab4f-5201977736a3/kube-storage-version-migrator-operator/0.log"
Oct 30 23:44:15.306981 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:15.306950    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-64cvm_b40f9c33-c4fe-4176-9ae9-561592a770f6/kube-multus-additional-cni-plugins/0.log"
Oct 30 23:44:15.337680 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:15.337654    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-64cvm_b40f9c33-c4fe-4176-9ae9-561592a770f6/egress-router-binary-copy/0.log"
Oct 30 23:44:15.379502 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:15.379475    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-64cvm_b40f9c33-c4fe-4176-9ae9-561592a770f6/cni-plugins/0.log"
Oct 30 23:44:15.417942 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:15.417911    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-64cvm_b40f9c33-c4fe-4176-9ae9-561592a770f6/bond-cni-plugin/0.log"
Oct 30 23:44:15.450601 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:15.450573    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-64cvm_b40f9c33-c4fe-4176-9ae9-561592a770f6/routeoverride-cni/0.log"
Oct 30 23:44:15.481858 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:15.481828    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-64cvm_b40f9c33-c4fe-4176-9ae9-561592a770f6/whereabouts-cni-bincopy/0.log"
Oct 30 23:44:15.512169 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:15.512139    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-64cvm_b40f9c33-c4fe-4176-9ae9-561592a770f6/whereabouts-cni/0.log"
Oct 30 23:44:16.101162 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:16.101125    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kfshl_87c8cdc5-be2e-47a0-8cde-e944fda51056/kube-multus/0.log"
Oct 30 23:44:16.385385 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:16.385299    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hmgsj_428997c4-b34e-4e3c-8551-c2d8510c69f4/network-metrics-daemon/0.log"
Oct 30 23:44:16.422590 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:16.422559    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hmgsj_428997c4-b34e-4e3c-8551-c2d8510c69f4/kube-rbac-proxy/0.log"
Oct 30 23:44:17.486758 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:17.486724    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ppff_9af08ecc-3199-4c8b-b364-f42f4befeff1/ovn-controller/0.log"
Oct 30 23:44:17.513343 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:17.513314    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ppff_9af08ecc-3199-4c8b-b364-f42f4befeff1/ovn-acl-logging/0.log"
Oct 30 23:44:17.551870 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:17.551840    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ppff_9af08ecc-3199-4c8b-b364-f42f4befeff1/kube-rbac-proxy-node/0.log"
Oct 30 23:44:17.601386 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:17.601362    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ppff_9af08ecc-3199-4c8b-b364-f42f4befeff1/kube-rbac-proxy-ovn-metrics/0.log"
Oct 30 23:44:17.641018 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:17.640993    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ppff_9af08ecc-3199-4c8b-b364-f42f4befeff1/northd/0.log"
Oct 30 23:44:17.679014 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:17.678987    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ppff_9af08ecc-3199-4c8b-b364-f42f4befeff1/nbdb/0.log"
Oct 30 23:44:17.712308 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:17.712277    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ppff_9af08ecc-3199-4c8b-b364-f42f4befeff1/sbdb/0.log"
Oct 30 23:44:17.795418 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:17.795339    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ppff_9af08ecc-3199-4c8b-b364-f42f4befeff1/ovnkube-controller/0.log"
Oct 30 23:44:20.351469 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:20.351441    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f78d59f99-5s5g6_53de1b04-dea0-4a6f-9609-760792cc0406/check-endpoints/0.log"
Oct 30 23:44:20.481479 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:20.481448    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-tgn5k_8eab1d96-88bb-417a-b03e-8e2f0274e862/network-check-target-container/0.log"
Oct 30 23:44:22.102684 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:22.102651    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-vhkmj_407cf470-1b82-424c-a8e2-a5f061b3b84c/iptables-alerter/0.log"
Oct 30 23:44:23.214762 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:23.214732    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-ggnd6_012d43c7-e3f2-4931-877e-cdf66eaf61b6/tuned/0.log"
Oct 30 23:44:25.633133 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:25.633098    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-r6fws_83a06c56-145a-4a52-baca-2d01c679e581/cluster-samples-operator/0.log"
Oct 30 23:44:25.684594 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:25.684562    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-r6fws_83a06c56-145a-4a52-baca-2d01c679e581/cluster-samples-operator-watch/0.log"
Oct 30 23:44:26.834119 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:26.834078    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-544d749c8c-2jl5h_9f16a8be-4c89-4786-a385-ff738661c9bb/service-ca-operator/1.log"
Oct 30 23:44:26.835025 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:26.835009    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-544d749c8c-2jl5h_9f16a8be-4c89-4786-a385-ff738661c9bb/service-ca-operator/0.log"
Oct 30 23:44:27.276900 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:27.276871    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-5fffc97bdf-qlkzv_afc45de8-4044-44c6-ac24-c39d8b2eece5/service-ca-controller/0.log"
Oct 30 23:44:27.815301 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:27.815269    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5z2tl_de0170e7-99d6-4102-9b88-13552cb6f421/csi-driver/0.log"
Oct 30 23:44:27.848264 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:27.848233    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5z2tl_de0170e7-99d6-4102-9b88-13552cb6f421/csi-node-driver-registrar/0.log"
Oct 30 23:44:27.879374 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:27.879340    5060 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5z2tl_de0170e7-99d6-4102-9b88-13552cb6f421/csi-liveness-probe/0.log"
Oct 30 23:44:34.774859 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:44:34.774821    5060 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6mwl7" containerName="extractor"
Oct 30 23:44:42.498790 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:42.498743    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"]
Oct 30 23:44:42.499342 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:42.498991    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" podUID="9411a073-738c-449d-904e-df96841920ae" containerName="istio-proxy" containerID="cri-o://8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8" gracePeriod=30
Oct 30 23:44:42.500151 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:42.500124    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"]
Oct 30 23:44:42.500396 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:42.500349    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" podUID="b79eadfc-8ca8-408a-9797-30a9d24dbdfa" containerName="istio-proxy" containerID="cri-o://a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d" gracePeriod=30
Oct 30 23:44:42.501314 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:42.501286    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"]
Oct 30 23:44:42.501487 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:42.501468    5060 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" podUID="d08b26c9-0d51-4613-9e2c-e37be3f1f026" containerName="istio-proxy" containerID="cri-o://7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606" gracePeriod=30
Oct 30 23:44:47.770028 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.769966    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:44:47.793405 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.793379    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:44:47.797383 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.797355    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 23:44:47.808870 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.808845    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9411a073-738c-449d-904e-df96841920ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9411a073-738c-449d-904e-df96841920ae-istiod-ca-cert\") pod \"9411a073-738c-449d-904e-df96841920ae\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") "
Oct 30 23:44:47.809002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.808882    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-kube-api-access-4q7wv\" (UniqueName: \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-kube-api-access-4q7wv\") pod \"9411a073-738c-449d-904e-df96841920ae\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") "
Oct 30 23:44:47.809002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.808915    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9411a073-738c-449d-904e-df96841920ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9411a073-738c-449d-904e-df96841920ae-istio-podinfo\") pod \"9411a073-738c-449d-904e-df96841920ae\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") "
Oct 30 23:44:47.809002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.808933    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-credential-socket\") pod \"9411a073-738c-449d-904e-df96841920ae\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") "
Oct 30 23:44:47.809002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.808990    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-socket\") pod \"9411a073-738c-449d-904e-df96841920ae\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") "
Oct 30 23:44:47.809231 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809054    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-data\") pod \"9411a073-738c-449d-904e-df96841920ae\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") "
Oct 30 23:44:47.809231 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809086    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-envoy\") pod \"9411a073-738c-449d-904e-df96841920ae\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") "
Oct 30 23:44:47.809231 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809113    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-certs\") pod \"9411a073-738c-449d-904e-df96841920ae\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") "
Oct 30 23:44:47.809231 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809174    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-istio-token\" (UniqueName: \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-istio-token\") pod \"9411a073-738c-449d-904e-df96841920ae\" (UID: \"9411a073-738c-449d-904e-df96841920ae\") "
Oct 30 23:44:47.809437 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809262    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-credential-socket" pod "9411a073-738c-449d-904e-df96841920ae" (UID: "9411a073-738c-449d-904e-df96841920ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.809437 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809269    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9411a073-738c-449d-904e-df96841920ae-istiod-ca-cert" pod "9411a073-738c-449d-904e-df96841920ae" (UID: "9411a073-738c-449d-904e-df96841920ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:47.810538 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809477    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-data" pod "9411a073-738c-449d-904e-df96841920ae" (UID: "9411a073-738c-449d-904e-df96841920ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.810538 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809555    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9411a073-738c-449d-904e-df96841920ae-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.810538 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809619    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.810538 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809679    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-socket" pod "9411a073-738c-449d-904e-df96841920ae" (UID: "9411a073-738c-449d-904e-df96841920ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.810538 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.809742    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-certs" pod "9411a073-738c-449d-904e-df96841920ae" (UID: "9411a073-738c-449d-904e-df96841920ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.811630 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.811594    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-kube-api-access-4q7wv" pod "9411a073-738c-449d-904e-df96841920ae" (UID: "9411a073-738c-449d-904e-df96841920ae"). InnerVolumeSpecName "kube-api-access-4q7wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:47.812006 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.811984    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-envoy" pod "9411a073-738c-449d-904e-df96841920ae" (UID: "9411a073-738c-449d-904e-df96841920ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.812006 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.812006    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-istio-token" pod "9411a073-738c-449d-904e-df96841920ae" (UID: "9411a073-738c-449d-904e-df96841920ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:47.812239 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.812215    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9411a073-738c-449d-904e-df96841920ae-istio-podinfo" pod "9411a073-738c-449d-904e-df96841920ae" (UID: "9411a073-738c-449d-904e-df96841920ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:44:47.910735 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910701    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-podinfo\") pod \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") "
Oct 30 23:44:47.910892 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910744    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-kube-api-access-mzvjd\" (UniqueName: \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-kube-api-access-mzvjd\") pod \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") "
Oct 30 23:44:47.910892 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910773    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-credential-socket\") pod \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") "
Oct 30 23:44:47.910892 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910799    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-envoy\") pod \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") "
Oct 30 23:44:47.910892 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910817    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istiod-ca-cert\") pod \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") "
Oct 30 23:44:47.910892 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910841    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-kube-api-access-f9tmx\" (UniqueName: \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-kube-api-access-f9tmx\") pod \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") "
Oct 30 23:44:47.910892 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910866    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-data\") pod \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910899    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-socket\") pod \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910929    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-certs\") pod \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910956    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-podinfo\") pod \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910982    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-envoy\") pod \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.910997    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-socket\") pod \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911032    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-token\" (UniqueName: \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-token\") pod \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911049    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-credential-socket\") pod \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911071    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istiod-ca-cert\") pod \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911130    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-token\" (UniqueName: \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-token\") pod \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911166    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-certs\") pod \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\" (UID: \"d08b26c9-0d51-4613-9e2c-e37be3f1f026\") "
Oct 30 23:44:47.911182 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911175    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-data" pod "d08b26c9-0d51-4613-9e2c-e37be3f1f026" (UID: "d08b26c9-0d51-4613-9e2c-e37be3f1f026"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911194    5060 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-data\") pod \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\" (UID: \"b79eadfc-8ca8-408a-9797-30a9d24dbdfa\") "
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911291    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-credential-socket" pod "b79eadfc-8ca8-408a-9797-30a9d24dbdfa" (UID: "b79eadfc-8ca8-408a-9797-30a9d24dbdfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911305    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istiod-ca-cert" pod "d08b26c9-0d51-4613-9e2c-e37be3f1f026" (UID: "d08b26c9-0d51-4613-9e2c-e37be3f1f026"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911431    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-data" pod "b79eadfc-8ca8-408a-9797-30a9d24dbdfa" (UID: "b79eadfc-8ca8-408a-9797-30a9d24dbdfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911526    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-socket" pod "d08b26c9-0d51-4613-9e2c-e37be3f1f026" (UID: "d08b26c9-0d51-4613-9e2c-e37be3f1f026"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911535    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-certs" pod "b79eadfc-8ca8-408a-9797-30a9d24dbdfa" (UID: "b79eadfc-8ca8-408a-9797-30a9d24dbdfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911633    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911654    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911668    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911717    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911731    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911745    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.911756 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911760    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.912409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911774    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.912409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911788    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q7wv\" (UniqueName: \"kubernetes.io/projected/9411a073-738c-449d-904e-df96841920ae-kube-api-access-4q7wv\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.912409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911806    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.912409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911820    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9411a073-738c-449d-904e-df96841920ae-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.912409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911834    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9411a073-738c-449d-904e-df96841920ae-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.912409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911841    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istiod-ca-cert" pod "b79eadfc-8ca8-408a-9797-30a9d24dbdfa" (UID: "b79eadfc-8ca8-408a-9797-30a9d24dbdfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:47.912409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911848    5060 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-data\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.912409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.911940    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-socket" pod "b79eadfc-8ca8-408a-9797-30a9d24dbdfa" (UID: "b79eadfc-8ca8-408a-9797-30a9d24dbdfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.912409 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.912091    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-credential-socket" pod "d08b26c9-0d51-4613-9e2c-e37be3f1f026" (UID: "d08b26c9-0d51-4613-9e2c-e37be3f1f026"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.913615 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.913585    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-podinfo" pod "b79eadfc-8ca8-408a-9797-30a9d24dbdfa" (UID: "b79eadfc-8ca8-408a-9797-30a9d24dbdfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:44:47.913806 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.913770    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-kube-api-access-f9tmx" pod "d08b26c9-0d51-4613-9e2c-e37be3f1f026" (UID: "d08b26c9-0d51-4613-9e2c-e37be3f1f026"). InnerVolumeSpecName "kube-api-access-f9tmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:47.913913 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.913808    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-certs" pod "d08b26c9-0d51-4613-9e2c-e37be3f1f026" (UID: "d08b26c9-0d51-4613-9e2c-e37be3f1f026"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.913913 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.913835    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-token" pod "d08b26c9-0d51-4613-9e2c-e37be3f1f026" (UID: "d08b26c9-0d51-4613-9e2c-e37be3f1f026"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:47.914089 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.914059    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-kube-api-access-mzvjd" pod "b79eadfc-8ca8-408a-9797-30a9d24dbdfa" (UID: "b79eadfc-8ca8-408a-9797-30a9d24dbdfa"). InnerVolumeSpecName "kube-api-access-mzvjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:47.914437 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.914412    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-podinfo" pod "d08b26c9-0d51-4613-9e2c-e37be3f1f026" (UID: "d08b26c9-0d51-4613-9e2c-e37be3f1f026"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:44:47.914437 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.914426    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-token" pod "b79eadfc-8ca8-408a-9797-30a9d24dbdfa" (UID: "b79eadfc-8ca8-408a-9797-30a9d24dbdfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:47.914655 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.914632    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-envoy" pod "b79eadfc-8ca8-408a-9797-30a9d24dbdfa" (UID: "b79eadfc-8ca8-408a-9797-30a9d24dbdfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.914788 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:47.914773    5060 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-envoy" pod "d08b26c9-0d51-4613-9e2c-e37be3f1f026" (UID: "d08b26c9-0d51-4613-9e2c-e37be3f1f026"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:48.012478 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012442    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012478 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012475    5060 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-workload-certs\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012703 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012490    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012703 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012502    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzvjd\" (UniqueName: \"kubernetes.io/projected/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-kube-api-access-mzvjd\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012703 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012540    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012703 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012554    5060 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9tmx\" (UniqueName: \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-kube-api-access-f9tmx\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012703 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012566    5060 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-workload-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012703 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012577    5060 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-podinfo\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012703 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012591    5060 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-envoy\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012703 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012603    5060 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08b26c9-0d51-4613-9e2c-e37be3f1f026-istio-token\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012703 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012615    5060 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08b26c9-0d51-4613-9e2c-e37be3f1f026-credential-socket\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.012703 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.012627    5060 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b79eadfc-8ca8-408a-9797-30a9d24dbdfa-istiod-ca-cert\") on node \"ip-10-0-138-251.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 23:44:48.395081 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.395045    5060 generic.go:356] "Generic (PLEG): container finished" podID="d08b26c9-0d51-4613-9e2c-e37be3f1f026" containerID="7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606" exitCode=0
Oct 30 23:44:48.395391 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.395085    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"
Oct 30 23:44:48.395391 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.395133    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" event={"ID":"d08b26c9-0d51-4613-9e2c-e37be3f1f026","Type":"ContainerDied","Data":"7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606"}
Oct 30 23:44:48.395391 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.395183    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5" event={"ID":"d08b26c9-0d51-4613-9e2c-e37be3f1f026","Type":"ContainerDied","Data":"606bd4a26bb24911fd33997e6c78104afe451ed90d9b53dc76e4bd8209f76625"}
Oct 30 23:44:48.395391 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.395205    5060 scope.go:122] "RemoveContainer" containerID="7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606"
Oct 30 23:44:48.396863 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.396838    5060 generic.go:356] "Generic (PLEG): container finished" podID="b79eadfc-8ca8-408a-9797-30a9d24dbdfa" containerID="a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d" exitCode=0
Oct 30 23:44:48.397002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.396876    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"
Oct 30 23:44:48.397002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.396911    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" event={"ID":"b79eadfc-8ca8-408a-9797-30a9d24dbdfa","Type":"ContainerDied","Data":"a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d"}
Oct 30 23:44:48.397002 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.396943    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c" event={"ID":"b79eadfc-8ca8-408a-9797-30a9d24dbdfa","Type":"ContainerDied","Data":"86a2654090cf0b83434bf638231a0dc2daeba7b83a122eb2cb73288e5fac5ccd"}
Oct 30 23:44:48.398327 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.398302    5060 generic.go:356] "Generic (PLEG): container finished" podID="9411a073-738c-449d-904e-df96841920ae" containerID="8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8" exitCode=0
Oct 30 23:44:48.398433 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.398348    5060 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"
Oct 30 23:44:48.398433 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.398352    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" event={"ID":"9411a073-738c-449d-904e-df96841920ae","Type":"ContainerDied","Data":"8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8"}
Oct 30 23:44:48.398433 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.398388    5060 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j" event={"ID":"9411a073-738c-449d-904e-df96841920ae","Type":"ContainerDied","Data":"aac4022a54a5b24aced2cae1ea3fa7f1eb7206e6cc5120f5e59422de12caaac5"}
Oct 30 23:44:48.404468 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.404450    5060 scope.go:122] "RemoveContainer" containerID="7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606"
Oct 30 23:44:48.404789 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:44:48.404767    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606\": container with ID starting with 7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606 not found: ID does not exist" containerID="7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606"
Oct 30 23:44:48.404863 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.404802    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606"} err="failed to get container status \"7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606\": rpc error: code = NotFound desc = could not find container \"7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606\": container with ID starting with 7fce37a2be2c0325f0a033fdb3163aab114df6d2c9db5aa10f8f7b139fe8b606 not found: ID does not exist"
Oct 30 23:44:48.404863 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.404826    5060 scope.go:122] "RemoveContainer" containerID="a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d"
Oct 30 23:44:48.414236 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.414213    5060 scope.go:122] "RemoveContainer" containerID="3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4"
Oct 30 23:44:48.421486 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.421467    5060 scope.go:122] "RemoveContainer" containerID="a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d"
Oct 30 23:44:48.421748 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:44:48.421728    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d\": container with ID starting with a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d not found: ID does not exist" containerID="a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d"
Oct 30 23:44:48.421809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.421755    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d"} err="failed to get container status \"a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d\": rpc error: code = NotFound desc = could not find container \"a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d\": container with ID starting with a14749590ee969d19fd605ec576946c7e6673ef753c8ba830ed8612de700112d not found: ID does not exist"
Oct 30 23:44:48.421809 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.421772    5060 scope.go:122] "RemoveContainer" containerID="3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4"
Oct 30 23:44:48.421983 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:44:48.421969    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4\": container with ID starting with 3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4 not found: ID does not exist" containerID="3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4"
Oct 30 23:44:48.422022 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.421986    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4"} err="failed to get container status \"3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4\": rpc error: code = NotFound desc = could not find container \"3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4\": container with ID starting with 3b1496a7154a492af71364b4e3eb632f0a58cee93c4b16a540bcb73776219ce4 not found: ID does not exist"
Oct 30 23:44:48.422022 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.422000    5060 scope.go:122] "RemoveContainer" containerID="8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8"
Oct 30 23:44:48.425298 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.425273    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"]
Oct 30 23:44:48.427998 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.427978    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-rd99c"]
Oct 30 23:44:48.430099 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.430069    5060 scope.go:122] "RemoveContainer" containerID="8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8"
Oct 30 23:44:48.430388 ip-10-0-138-251 kubenswrapper[5060]: E1030 23:44:48.430368    5060 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8\": container with ID starting with 8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8 not found: ID does not exist" containerID="8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8"
Oct 30 23:44:48.430481 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.430391    5060 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8"} err="failed to get container status \"8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8\": rpc error: code = NotFound desc = could not find container \"8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8\": container with ID starting with 8d9a27f9a30f5a1e361506ee3a96a0ff63db20d64d7b1f78a72148a0734c94f8 not found: ID does not exist"
Oct 30 23:44:48.442387 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.442366    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"]
Oct 30 23:44:48.448388 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.448366    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-bv98j"]
Oct 30 23:44:48.466758 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.466733    5060 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"]
Oct 30 23:44:48.473208 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:48.473186    5060 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-86b669b864-zctx5"]
Oct 30 23:44:49.289630 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:49.289600    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9411a073-738c-449d-904e-df96841920ae" path="/var/lib/kubelet/pods/9411a073-738c-449d-904e-df96841920ae/volumes"
Oct 30 23:44:49.290070 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:49.290054    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79eadfc-8ca8-408a-9797-30a9d24dbdfa" path="/var/lib/kubelet/pods/b79eadfc-8ca8-408a-9797-30a9d24dbdfa/volumes"
Oct 30 23:44:49.290494 ip-10-0-138-251 kubenswrapper[5060]: I1030 23:44:49.290482    5060 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d08b26c9-0d51-4613-9e2c-e37be3f1f026" path="/var/lib/kubelet/pods/d08b26c9-0d51-4613-9e2c-e37be3f1f026/volumes"