Oct 30 02:54:55.757291 ip-10-0-130-130 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:54:56.171853 ip-10-0-130-130 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:56.171853 ip-10-0-130-130 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:54:56.171853 ip-10-0-130-130 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:56.171853 ip-10-0-130-130 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:56.177354 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177258    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:56.177354 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177353    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:56.177354 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177359    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177362    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177365    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177368    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177371    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177374    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177377    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177379    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177382    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177384    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177387    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177389    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177392    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177394    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177397    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177399    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177406    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177409    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177412    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177415    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:56.177451 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177417    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177420    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177423    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177426    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177428    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177431    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177433    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177436    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177438    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177441    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177444    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177446    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177449    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177451    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177455    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177458    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177461    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177464    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177466    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177469    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:56.178038 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177471    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177475    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177478    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177480    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177483    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177485    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177488    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177490    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177493    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177497    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177499    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177502    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177504    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177507    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177509    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177514    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177518    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177521    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177524    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:56.178523 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177526    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177529    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177532    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177534    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177537    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177539    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177543    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177546    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177549    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177552    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177554    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177557    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177560    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177562    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177565    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177567    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177570    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177573    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177575    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177578    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:56.178983 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177581    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177583    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177586    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177589    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177591    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177594    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177596    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177599    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177601    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177603    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177606    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177608    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177612    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177616    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177620    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177623    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177625    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177628    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177631    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:56.179461 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177634    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177636    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177639    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177641    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177644    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.177647    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178534    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178543    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178545    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178548    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178551    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178553    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178557    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178560    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178562    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178565    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178568    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178571    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178573    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:56.179930 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178576    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178579    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178583    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178585    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178588    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178591    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178593    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178596    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178598    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178601    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178603    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178606    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178609    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178611    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178613    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178616    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178618    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178621    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178623    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178626    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:56.180394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178628    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178631    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178633    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178635    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178638    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178641    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178643    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178645    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178648    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178651    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178653    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178657    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178659    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178662    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178664    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178667    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178672    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178675    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178677    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:56.180889 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178679    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178682    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178684    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178687    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178690    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178692    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178695    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178697    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178700    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178702    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178705    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178707    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178710    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178712    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178714    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178717    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178720    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178724    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178728    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:56.181394 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178731    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178734    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178737    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178740    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178743    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178746    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178749    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178752    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178755    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178758    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178762    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178764    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178767    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178769    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178772    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178774    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178777    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178779    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178782    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178784    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:56.181846 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178787    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178789    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178791    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178794    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178796    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178799    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178802    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178804    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178807    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178811    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178814    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178818    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178821    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178824    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.178827    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179386    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179393    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179398    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179402    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:56.182323 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179405    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179408    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179412    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179415    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179418    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179420    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179423    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179426    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179428    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179430    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179433    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179436    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179439    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179441    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179444    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179446    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179449    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179452    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179454    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:56.182813 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179457    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179459    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179461    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179464    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179467    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179469    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179472    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179475    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179477    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179480    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179483    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179485    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179488    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179491    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179494    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179496    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179499    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179502    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179504    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179506    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:56.183284 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179509    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179512    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179514    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179517    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179519    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179522    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179524    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179527    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179529    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179532    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179534    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179536    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179539    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179541    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179543    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179546    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179548    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179551    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179553    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:56.183784 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179555    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179558    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179560    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179563    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179565    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179568    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179570    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179573    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179578    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179582    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179585    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179588    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179590    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179593    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179595    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179598    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179600    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179603    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179605    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179608    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:56.184253 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179610    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179613    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179615    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179618    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179620    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179623    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179625    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179628    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179630    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179633    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179635    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179637    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179640    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179642    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179645    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179648    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179652    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179654    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179657    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179659    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:56.184724 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179662    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179665    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179668    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.179670    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181568    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181585    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181591    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181597    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181601    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181605    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181609    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181614    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181617    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181620    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181624    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181627    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181630    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181634    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181637    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181640    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181643    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181646    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181651    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181654    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:54:56.185223 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181658    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181661    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181664    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181669    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181672    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181675    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181679    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181682    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181685    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181691    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181694    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181698    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181703    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181706    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181709    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181712    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181716    2580 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181719    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181723    2580 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181727    2580 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181730    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181733    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181736    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181740    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:54:56.185797 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181743    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181746    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181750    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181753    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181756    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181759    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181762    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181765    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181768    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181770    2580 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181776    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181780    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181783    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181787    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181790    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181793    2580 flags.go:64] FLAG: --help="false"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181796    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181800    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181804    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181807    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181811    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181814    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181817    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:54:56.186401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181820    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181823    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181826    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181829    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181832    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181835    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181838    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181841    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181844    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181847    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181850    2580 flags.go:64] FLAG: --lock-file=""
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181853    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181856    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181859    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181865    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181868    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181871    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181874    2580 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181877    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181881    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181884    2580 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181887    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181892    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181906    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181911    2580 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:54:56.186980 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181914    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181918    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181920    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181924    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181927    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181930    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181933    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181941    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181944    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181947    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181950    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181953    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181956    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181959    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181961    2580 flags.go:64] FLAG: --port="10250"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181965    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181968    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f104715efb0af7e7"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181971    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181973    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181976    2580 flags.go:64] FLAG: --register-node="true"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181979    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181983    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181986    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181989    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181992    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181995    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:54:56.187615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.181999    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182002    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182005    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182007    2580 flags.go:64] FLAG: --runonce="false"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182010    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182013    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182016    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182019    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182022    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182025    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182029    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182032    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182035    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182038    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182041    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182044    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182047    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182050    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182053    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182059    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182061    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182064    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182069    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182072    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182074    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:54:56.188253 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182078    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182081    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182084    2580 flags.go:64] FLAG: --v="2"
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182089    2580 flags.go:64] FLAG: --version="false"
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182093    2580 flags.go:64] FLAG: --vmodule=""
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182098    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182101    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182243    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182247    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182249    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182252    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182255    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182257    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182260    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182263    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182266    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182268    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182270    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182273    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182276    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:56.188911 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182279    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182282    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182284    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182287    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182289    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182292    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182294    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182297    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182299    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182302    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182304    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182307    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182309    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182312    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182314    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182317    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182320    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182322    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182326    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182329    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:56.189436 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182331    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182334    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182337    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182339    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182342    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182344    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182347    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182349    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182352    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182354    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182357    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182359    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182362    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182366    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182369    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182372    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182374    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182377    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182379    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:56.190003 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182382    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182384    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182388    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182391    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182393    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182396    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182399    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182401    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182404    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182406    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182410    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182413    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182416    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182418    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182421    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182424    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182426    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182429    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182431    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182433    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:56.190640 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182436    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182438    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182441    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182443    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182446    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182449    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182452    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182454    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182457    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182459    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182462    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182464    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182466    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182470    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182476    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182479    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182482    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182485    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182487    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182490    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:56.191380 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182492    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182495    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182499    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182501    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182504    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182506    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182509    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182511    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182514    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182516    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182519    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182521    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182523    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182526    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.182557    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182601    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:56.191885 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182606    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182609    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182613    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182616    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182618    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182621    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182624    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182626    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182629    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182631    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182634    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182638    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182641    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182643    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182646    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182648    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182651    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182654    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182658    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182660    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:56.192294 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182663    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182665    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182668    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182671    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182673    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182676    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182678    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182680    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182683    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182685    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182688    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182690    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182693    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182696    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182698    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182701    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182704    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182706    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182709    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:56.192790 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182711    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182714    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182716    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182719    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182723    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182727    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182730    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182733    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182735    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182738    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182741    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182745    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182747    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182750    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182752    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182755    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182757    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182760    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182762    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:56.193302 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182765    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182768    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182770    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182773    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182775    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182778    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182780    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182783    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182785    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182788    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182791    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182793    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182795    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182798    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182800    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182803    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182805    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182810    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182812    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182815    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:56.193752 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182818    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182820    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182823    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182825    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182829    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182834    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182838    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182842    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182845    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182848    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182850    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182853    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182856    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182859    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182861    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182863    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182866    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182868    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182871    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182873    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:56.194433 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182875    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182878    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182880    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182883    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182886    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182888    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.182890    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.191963    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.191985    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.192757    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.192764    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192839    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192845    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192848    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192851    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192855    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192857    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192860    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192863    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:56.194941 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192867    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192873    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192876    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192878    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192881    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192884    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192887    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192889    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192892    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192911    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192916    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192921    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192925    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192929    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192931    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192936    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192939    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192942    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192945    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192947    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:56.195404 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192951    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192954    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192956    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192959    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192962    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192965    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192967    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192970    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192972    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192976    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192980    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192984    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192987    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192989    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192992    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192995    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.192998    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193001    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193004    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:56.195870 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193007    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193010    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193012    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193015    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193018    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193020    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193023    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193025    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193028    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193031    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193034    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193036    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193039    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193042    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193045    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193047    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193050    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193052    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193055    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193057    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:56.196335 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193060    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193063    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193065    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193068    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193071    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193073    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193076    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193078    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193080    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193083    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193085    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193088    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193091    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193093    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193096    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193098    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193101    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193103    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193106    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193108    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:56.196831 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193111    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193115    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193117    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193119    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193122    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193125    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193127    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193130    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193132    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193135    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193137    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193139    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193142    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193144    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193146    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193149    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193151    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193154    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:56.197313 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193156    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.193188    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193329    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193334    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193337    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193340    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193343    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193346    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193349    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193351    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193354    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193356    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193359    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193361    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193364    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193366    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:56.197751 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193369    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193372    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193374    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193377    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193380    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193383    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193386    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193388    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193391    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193393    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193395    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193398    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193401    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193403    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193405    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193408    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193411    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193413    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193415    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193419    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:56.198152 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193421    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193424    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193426    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193429    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193432    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193435    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193437    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193440    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193443    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193445    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193448    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193450    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193453    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193456    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193459    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193464    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193467    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193470    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193473    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:56.198642 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193475    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193478    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193481    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193483    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193486    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193488    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193491    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193493    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193496    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193498    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193501    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193503    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193505    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193508    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193510    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193513    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193515    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193517    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193521    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:56.199216 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193524    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193526    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193529    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193531    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193534    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193536    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193539    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193541    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193544    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193547    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193550    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193552    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193555    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193558    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193560    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193564    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193567    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193571    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193574    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:56.199675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193576    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193579    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193581    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193584    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193586    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193589    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193591    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193593    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193596    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193598    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193601    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193603    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193605    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193608    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193611    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:56.200131 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.193635    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193668    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193672    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193675    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193678    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193682    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193685    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193688    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193691    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193694    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193697    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193699    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193702    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193704    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193707    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193710    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193712    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193714    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193717    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193719    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:56.200501 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193722    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193724    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193727    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193729    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193732    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193734    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193737    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193739    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193742    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193745    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193747    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193750    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193754    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193756    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193759    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193762    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193765    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193767    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193770    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193772    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:56.201008 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193775    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193777    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193780    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193782    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193785    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193787    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193790    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193793    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193795    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193798    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193800    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193804    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193808    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193812    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193815    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193817    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193820    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193822    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193825    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:56.201487 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193827    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193830    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193832    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193837    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193840    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193843    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193846    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193849    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193851    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193854    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193857    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193859    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193862    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193864    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193867    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193869    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193872    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193874    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193878    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:56.201986 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193880    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193883    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193885    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193888    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193890    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193893    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193911    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193918    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193920    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193923    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193926    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193929    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193933    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193936    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193940    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193943    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193951    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193954    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193956    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:56.202445 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193960    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193963    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193967    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193970    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193973    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193975    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193978    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193981    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193983    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:56.193985    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.194054    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.196556    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.197696    2580 server.go:1014] "Starting client certificate rotation"
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.197793    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.197829    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.200032    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:54:56.202995 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.201299    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:54:56.212614 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.212593    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:54:56.216765 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.216749    2580 log.go:25] "Validated CRI v1 image API"
Oct 30 02:54:56.217716 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.217689    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:54:56.221320 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.221284    2580 fs.go:135] Filesystem UUIDs: map[44c5f80b-4751-4159-93bc-0d264142889d:/dev/nvme0n1p3 49b7a8ae-bb84-4666-9864-724fa5d7fc51:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 02:54:56.221429 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.221316    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 02:54:56.228066 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.227924    2580 manager.go:217] Machine: {Timestamp:2026-10-30 02:54:56.226443554 +0000 UTC m=+0.364049157 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100236 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ae4d5b6a527d0b677103a0ad28fa6 SystemUUID:ec2ae4d5-b6a5-27d0-b677-103a0ad28fa6 BootID:53a0ee35-d5e0-4490-a2b9-ecd29de1b4ed Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:5e:09:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:5e:09:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:cc:44:74:6e:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:54:56.228066 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.228056    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:54:56.228266 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.228251    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:54:56.231876 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.231838    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:54:56.232074 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.231876    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:54:56.232173 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.232080    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:54:56.232173 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.232092    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:54:56.232173 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.232155    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:54:56.232282 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.232177    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:54:56.233454 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.233439    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:54:56.238186 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.238168    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:54:56.238343 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.238332    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:54:56.239205 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.239194    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:54:56.239273 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.239215    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:54:56.239273 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.239232    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:54:56.239273 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.239246    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:54:56.239392 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.239283    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:54:56.239666 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.239651    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:54:56.243657 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.243637    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:54:56.250368 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.250354    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:54:56.251267 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.251255    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:54:56.251313 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.251285    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:54:56.251447 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.251437    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:54:56.251478 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.251453    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:54:56.251478 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.251460    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:54:56.251478 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.251466    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:54:56.251478 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.251471    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:54:56.251478 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.251477    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:54:56.252290 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.252281    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:54:56.252327 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.252291    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:54:56.252327 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.252298    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:54:56.252327 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.252304    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:54:56.252327 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.252313    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:54:56.252327 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.252321    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:54:56.253050 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.253041    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:54:56.253087 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.253051    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:54:56.257245 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.257230    2580 server.go:1263] "Started kubelet"
Oct 30 02:54:56.257395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.257328    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:54:56.257490 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.257385    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:54:56.257490 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.257460    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:54:56.258176 ip-10-0-130-130 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:54:56.259346 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.259148    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:54:56.260330 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.260305    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:54:56.262970 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.262951    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 02:54:56.266482 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.266464    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:54:56.267855 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.267841    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:54:56.268022 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.267974    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:56.268656 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.268631    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:54:56.268656 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.268630    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:54:56.268656 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.268657    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:54:56.268812 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.268751    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:54:56.268812 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.268759    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:54:56.269102 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.269085    2580 factory.go:153] Registering CRI-O factory
Oct 30 02:54:56.269177 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.269102    2580 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:54:56.269177 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.269157    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:54:56.269177 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.269166    2580 factory.go:55] Registering systemd factory
Oct 30 02:54:56.269177 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.269175    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:54:56.269619 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.269193    2580 factory.go:103] Registering Raw factory
Oct 30 02:54:56.269619 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.269207    2580 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:54:56.270324 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.270309    2580 manager.go:319] Starting recovery of all containers
Oct 30 02:54:56.281863 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.281832    2580 manager.go:324] Recovery completed
Oct 30 02:54:56.286865 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.286840    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-130.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 02:54:56.286979 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.286851    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 02:54:56.286979 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.286971    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:56.287621 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.287598    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:54:56.287695 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.287676    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4khkj"
Oct 30 02:54:56.287954 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.286846    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-130.ec2.internal.18e32f2132da8fb3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-130.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-130.ec2.internal,},FirstTimestamp:2026-10-30 02:54:56.257200051 +0000 UTC m=+0.394805653,LastTimestamp:2026-10-30 02:54:56.257200051 +0000 UTC m=+0.394805653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-130.ec2.internal,}"
Oct 30 02:54:56.289057 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.289041    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:56.289109 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.289072    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:56.289109 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.289083    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:56.289710 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.289698    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:54:56.289710 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.289708    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:54:56.289793 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.289710    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:56.289793 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.289723    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:54:56.290414 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.290398    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:56.290501 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.290425    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:56.290501 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.290441    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:56.291932 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.291921    2580 policy_none.go:50] "Start"
Oct 30 02:54:56.291983 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.291937    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:54:56.291983 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.291951    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:54:56.292358 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.292340    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4khkj"
Oct 30 02:54:56.293939 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.293927    2580 policy_none.go:44] "Start"
Oct 30 02:54:56.306536 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.306369    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-130.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 02:54:56.306536 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.306398    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 02:54:56.306673 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.306553    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 02:54:56.306731 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.306633    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-130.ec2.internal.18e32f2134c0aba7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-130.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-130.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-130.ec2.internal,},FirstTimestamp:2026-10-30 02:54:56.289057703 +0000 UTC m=+0.426663305,LastTimestamp:2026-10-30 02:54:56.289057703 +0000 UTC m=+0.426663305,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-130.ec2.internal,}"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.341629    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.346055    2580 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.346107    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.346120    2580 server.go:87] "Starting device plugin registration server"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.346405    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.346417    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.346519    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.346592    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.346601    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.347215    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.347264    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.350200    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.351565    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.351585    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.351603    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:54:56.352682 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.351635    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 02:54:56.368352 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.368321    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:56.447568 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.447483    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:56.448550 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.448530    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:56.448661 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.448565    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:56.448661 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.448579    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:56.448661 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.448636    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.452628 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.452607    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"]
Oct 30 02:54:56.452718 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.452676    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:56.453405 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.453391    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:56.453473 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.453432    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:56.453473 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.453443    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:56.455679 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.455667    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:56.455863 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.455848    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.455933 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.455889    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:56.456423 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.456406    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:56.456497 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.456440    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:56.456497 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.456412    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:56.456497 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.456486    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:56.456497 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.456459    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:56.456645 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.456511    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:56.458772 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.458756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.458836 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.458783    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:56.459655 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.459640    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:56.459736 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.459670    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:56.459736 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.459683    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:56.465770 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.465754    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.465833 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.465778    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-130.ec2.internal\": node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:56.480391 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.480374    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-130.ec2.internal\" not found" node="ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.482391 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.482375    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-130.ec2.internal\" not found" node="ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.505707 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.505677    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:56.570063 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.570025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.570063 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.570060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.570265 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.570084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/868aae1459799721b266e5e297e80105-config\") pod \"kube-apiserver-proxy-ip-10-0-130-130.ec2.internal\" (UID: \"868aae1459799721b266e5e297e80105\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.606055 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.606023    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:56.671244 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.671212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.671244 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.671245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.671443 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.671269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/868aae1459799721b266e5e297e80105-config\") pod \"kube-apiserver-proxy-ip-10-0-130-130.ec2.internal\" (UID: \"868aae1459799721b266e5e297e80105\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.671443 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.671352    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.671443 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.671357    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.671443 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.671355    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/868aae1459799721b266e5e297e80105-config\") pod \"kube-apiserver-proxy-ip-10-0-130-130.ec2.internal\" (UID: \"868aae1459799721b266e5e297e80105\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.706456 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.706381    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:56.785298 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.785258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.788425 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:56.788403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:56.806932 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.806884    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:56.907468 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:56.907414    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:57.008105 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:57.008009    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:57.108759 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:57.108724    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:57.197232 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.197201    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 02:54:57.197725 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.197395    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:54:57.209534 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:57.209506    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:57.287849 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.287772    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:57.297883 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.297830    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:49:56 +0000 UTC" deadline="2028-05-23 18:40:17.629043446 +0000 UTC"
Oct 30 02:54:57.297883 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.297883    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13719h45m20.331163751s"
Oct 30 02:54:57.310650 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:57.310614    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:57.352034 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:57.351996    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868aae1459799721b266e5e297e80105.slice/crio-16fe60e79e0c3502099ee29d15fd36b92eb474340d9795b90ce55e94945812fc WatchSource:0}: Error finding container 16fe60e79e0c3502099ee29d15fd36b92eb474340d9795b90ce55e94945812fc: Status 404 returned error can't find the container with id 16fe60e79e0c3502099ee29d15fd36b92eb474340d9795b90ce55e94945812fc
Oct 30 02:54:57.357407 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:57.357375    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2464e4e1bac4126b7f6dfa935cfd24ca.slice/crio-b47016b6f5df54ae2183eb9ae086099c839f58a485f50abed70866ca5bb50fd4 WatchSource:0}: Error finding container b47016b6f5df54ae2183eb9ae086099c839f58a485f50abed70866ca5bb50fd4: Status 404 returned error can't find the container with id b47016b6f5df54ae2183eb9ae086099c839f58a485f50abed70866ca5bb50fd4
Oct 30 02:54:57.358185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.358168    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:54:57.385657 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.385623    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:57.386908 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.386878    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:57.387000 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.386935    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:57.387000 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.386947    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:57.411290 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:57.411259    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:54:57.510148 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.510112    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:57.568519 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.568406    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:57.593125 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.593092    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:57.595168 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.595149    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:54:57.616438 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.616411    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:57.938414 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:57.938305    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:58.058605 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.058574    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:58.240693 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.240615    2580 apiserver.go:52] "Watching apiserver"
Oct 30 02:54:58.267795 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.267760    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 02:54:58.274130 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.274096    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:54:58.274613 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.274576    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal","openshift-image-registry/node-ca-f8mgk","kube-system/konnectivity-agent-m2tdr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t","openshift-multus/multus-g5vjw","openshift-multus/network-metrics-daemon-ndbtf","openshift-network-diagnostics/network-check-target-rjwmx","kube-system/global-pull-secret-syncer-ddlpl","openshift-dns/node-resolver-vsx5v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal","openshift-multus/multus-additional-cni-plugins-cspkf","openshift-cluster-node-tuning-operator/tuned-p6dt8","openshift-network-operator/iptables-alerter-6j76z","openshift-ovn-kubernetes/ovnkube-node-xltm9"]
Oct 30 02:54:58.277990 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.277928    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:54:58.278130 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.278041    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:54:58.280145 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.280119    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.282929 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.282908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:54:58.284999 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.284976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.287701 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.287674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.289888 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.289866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:54:58.290018 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.289963    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:54:58.292845 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.292826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.297261 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.297235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.298264 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.298225    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:58.298364 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.298290    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:58.298364 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.298321    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9qpgh\""
Oct 30 02:54:58.298364 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.298328    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:54:58.299927 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.299850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.301777 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.301755    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:54:58.301884 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.301874    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:54:58.302129 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.302037    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mk2hv\""
Oct 30 02:54:58.302129 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.302100    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:54:58.302798 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.302683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:54:58.302798 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.302761    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:54:58.303197 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.303136    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:54:58.303197 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.303146    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:54:58.303197 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.303171    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8ljf8\""
Oct 30 02:54:58.304641 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.304607    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:54:58.305179 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.305160    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:54:58.305265 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.305203    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:54:58.305324 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.305282    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ft4zm\""
Oct 30 02:54:58.305380 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.305339    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:54:58.305834 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.305812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.308675 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.308638    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.310957 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.309891    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gpqv\""
Oct 30 02:54:58.310957 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.309937    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:54:58.310957 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.309997    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:54:58.315658 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.315633    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:54:58.315765 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.315681    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:54:58.315765 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.315723    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:54:58.317480 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.317058    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fdxtg\""
Oct 30 02:54:58.317480 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.317137    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:54:58.317480 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.317282    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:54:58.317480 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.317438    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:54:58.320848 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.320827    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-chr82\""
Oct 30 02:54:58.322995 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.322958    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:54:58.323098 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.323034    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pkk6c"
Oct 30 02:54:58.331859 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.331829    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pkk6c"
Oct 30 02:54:58.358746 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.358692    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal" event={"ID":"2464e4e1bac4126b7f6dfa935cfd24ca","Type":"ContainerStarted","Data":"b47016b6f5df54ae2183eb9ae086099c839f58a485f50abed70866ca5bb50fd4"}
Oct 30 02:54:58.360328 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.360269    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal" event={"ID":"868aae1459799721b266e5e297e80105","Type":"ContainerStarted","Data":"16fe60e79e0c3502099ee29d15fd36b92eb474340d9795b90ce55e94945812fc"}
Oct 30 02:54:58.370484 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.370453    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:54:58.382603 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.382558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac850265-e643-4365-b318-3507b8c838b9-hosts-file\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.382757 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.382621    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b86p\" (UniqueName: \"kubernetes.io/projected/27811319-29a6-459b-a1e3-8325c6cc2274-kube-api-access-2b86p\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.382757 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.382739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-systemd\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.382866 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.382779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cni-binary-copy\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.382866 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.382808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-serviceca\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.382954 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.382863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.382954 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.382925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.383058 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.382973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.383058 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-registration-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.383152 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.383152 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-kubernetes\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.383152 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:54:58.383274 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-dbus\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:54:58.383274 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-sys\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.383274 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-netns\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.383274 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-kubelet-config\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:54:58.383427 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.383427 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac850265-e643-4365-b318-3507b8c838b9-tmp-dir\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.383427 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-systemd-units\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.383427 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-ovn\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.383427 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383379    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-lib-modules\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.383427 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-var-lib-kubelet\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.383427 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-etc-kubernetes\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.383708 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.383708 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-node-log\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.383708 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383521    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-config\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.383708 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-env-overrides\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.383708 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5txc\" (UniqueName: \"kubernetes.io/projected/1db5896f-bc1f-4965-9282-a344f5ad8e03-kube-api-access-b5txc\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.383708 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cnibin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.383708 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-conf-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.383708 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383641    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-netd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.383708 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-system-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-multus-certs\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-script-lib\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-modprobe-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysconfig\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-host-slash\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383845    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-daemon-config\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr48h\" (UniqueName: \"kubernetes.io/projected/2a73c32c-6071-4eb7-8215-171cb490ee56-kube-api-access-xr48h\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.383989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-cnibin\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-device-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-etc-selinux\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.384093 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-tuned\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-hostroot\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-etc-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-bin\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-socket-dir-parent\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4tt\" (UniqueName: \"kubernetes.io/projected/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-kube-api-access-9m4tt\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384265    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-slash\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-log-socket\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-tmp\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-agent-certs\") pod \"konnectivity-agent-m2tdr\" (UID: \"c8c74de0-5c52-4e2e-9f03-3b982a4e7667\") " pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfjxv\" (UniqueName: \"kubernetes.io/projected/ac850265-e643-4365-b318-3507b8c838b9-kube-api-access-tfjxv\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-kubelet\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-k8s-cni-cncf-io\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-multus\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.384557 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-host\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-socket-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-sys-fs\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nflm\" (UniqueName: \"kubernetes.io/projected/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-kube-api-access-4nflm\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2hwt\" (UniqueName: \"kubernetes.io/projected/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-kube-api-access-d2hwt\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjv7m\" (UniqueName: \"kubernetes.io/projected/603f9058-eea9-4899-872c-503c7111dcc7-kube-api-access-tjv7m\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-systemd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-var-lib-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-iptables-alerter-script\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-kubelet\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-system-cni-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-host\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-bin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbnn8\" (UniqueName: \"kubernetes.io/projected/a1df94c2-8be5-42df-9d04-7569e45f7648-kube-api-access-kbnn8\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-netns\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.385185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27811319-29a6-459b-a1e3-8325c6cc2274-ovn-node-metrics-cert\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.385756 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-conf\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.385756 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-run\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.385756 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.384959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-konnectivity-ca\") pod \"konnectivity-agent-m2tdr\" (UID: \"c8c74de0-5c52-4e2e-9f03-3b982a4e7667\") " pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:54:58.410450 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.410420    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:54:58.410653 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.410503    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:54:58.410653 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.410552    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:54:58.410746 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.410706    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:54:58.411000 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.410965    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kn7z4\""
Oct 30 02:54:58.411000 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.410998    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:58.411210 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.410999    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:58.411210 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.411029    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:54:58.411210 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.411058    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tpj88\""
Oct 30 02:54:58.485920 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.485870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-env-overrides\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.486116 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.485944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5txc\" (UniqueName: \"kubernetes.io/projected/1db5896f-bc1f-4965-9282-a344f5ad8e03-kube-api-access-b5txc\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.486116 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.485979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cnibin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.486116 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-conf-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.486116 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-netd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.486116 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-system-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.486116 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-multus-certs\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.486116 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-script-lib\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-modprobe-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysconfig\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-host-slash\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-daemon-config\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr48h\" (UniqueName: \"kubernetes.io/projected/2a73c32c-6071-4eb7-8215-171cb490ee56-kube-api-access-xr48h\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-cnibin\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-device-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-etc-selinux\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486367    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.486395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-tuned\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-hostroot\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-etc-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-bin\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-socket-dir-parent\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4tt\" (UniqueName: \"kubernetes.io/projected/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-kube-api-access-9m4tt\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486622    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-slash\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-log-socket\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-tmp\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-agent-certs\") pod \"konnectivity-agent-m2tdr\" (UID: \"c8c74de0-5c52-4e2e-9f03-3b982a4e7667\") " pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfjxv\" (UniqueName: \"kubernetes.io/projected/ac850265-e643-4365-b318-3507b8c838b9-kube-api-access-tfjxv\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-kubelet\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-k8s-cni-cncf-io\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-multus\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.486816 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-host\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-socket-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-sys-fs\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nflm\" (UniqueName: \"kubernetes.io/projected/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-kube-api-access-4nflm\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2hwt\" (UniqueName: \"kubernetes.io/projected/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-kube-api-access-d2hwt\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486916    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjv7m\" (UniqueName: \"kubernetes.io/projected/603f9058-eea9-4899-872c-503c7111dcc7-kube-api-access-tjv7m\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.486968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-systemd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-var-lib-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-iptables-alerter-script\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-kubelet\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-system-cni-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487098    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-host\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-bin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbnn8\" (UniqueName: \"kubernetes.io/projected/a1df94c2-8be5-42df-9d04-7569e45f7648-kube-api-access-kbnn8\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-netns\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27811319-29a6-459b-a1e3-8325c6cc2274-ovn-node-metrics-cert\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.487590 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-conf\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-run\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-konnectivity-ca\") pod \"konnectivity-agent-m2tdr\" (UID: \"c8c74de0-5c52-4e2e-9f03-3b982a4e7667\") " pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac850265-e643-4365-b318-3507b8c838b9-hosts-file\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b86p\" (UniqueName: \"kubernetes.io/projected/27811319-29a6-459b-a1e3-8325c6cc2274-kube-api-access-2b86p\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-systemd\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cni-binary-copy\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-serviceca\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-registration-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-kubernetes\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-dbus\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-sys\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.488242 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-netns\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-kubelet-config\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac850265-e643-4365-b318-3507b8c838b9-tmp-dir\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487788    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-systemd-units\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-ovn\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-lib-modules\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-var-lib-kubelet\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-etc-kubernetes\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-node-log\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.487961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-config\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.488909 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.488684    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-config\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.489429 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.489149    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-env-overrides\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.489594 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.489572    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cnibin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.489676 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.489634    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-conf-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.489734 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.489679    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-netd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.489787 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.489736    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-system-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.489787 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.489772    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-multus-certs\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.490302 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.490281    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-script-lib\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.490440 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.490417    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-modprobe-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.490499 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.490482    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysconfig\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.490542 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.490525    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-host-slash\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.491101 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.491030    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-daemon-config\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.491376 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.491355    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-cnibin\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.491453 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.491414    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.491515 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.491473    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-device-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.491574 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.491528    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-bin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.491631 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.491575    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-host\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.491631 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.491595    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-etc-selinux\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.491742 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.491715    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.491868 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.491840    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-netns\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.492068 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.492044    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:54:58.492248 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.492201    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.492317 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.492306    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-hostroot\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.492395 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.492364    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-etc-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.492482 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.492417    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-bin\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.492570 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.492550    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-socket-dir-parent\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.492999 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.492979    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.494312 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.493737    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-kubernetes\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.494312 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.493961    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-conf\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.494312 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.494039    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-run\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.494312 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.494195    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac850265-e643-4365-b318-3507b8c838b9-tmp-dir\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.494496 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.494324    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:58.494496 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.494397    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs podName:2a73c32c-6071-4eb7-8215-171cb490ee56 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.994374117 +0000 UTC m=+3.131979735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs") pod "network-metrics-daemon-ndbtf" (UID: "2a73c32c-6071-4eb7-8215-171cb490ee56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:58.494693 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.494671    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-dbus\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:54:58.494762 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.494746    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-sys\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.494812 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.494798    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-netns\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.494868 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.494853    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-kubelet-config\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:54:58.494925 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.494911    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.495003 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.494968    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-slash\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.495426 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.495402    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-konnectivity-ca\") pod \"konnectivity-agent-m2tdr\" (UID: \"c8c74de0-5c52-4e2e-9f03-3b982a4e7667\") " pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:54:58.495539 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.495519    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-serviceca\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.495656 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.495639    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac850265-e643-4365-b318-3507b8c838b9-hosts-file\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.495966 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.495946    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.496201 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496180    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-systemd\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.496562 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496525    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.496673 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496597    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.496673 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496635    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-registration-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.496673 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496635    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27811319-29a6-459b-a1e3-8325c6cc2274-ovn-node-metrics-cert\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.496673 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496664    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496727    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-kubelet\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496784    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-systemd-units\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496787    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-agent-certs\") pod \"konnectivity-agent-m2tdr\" (UID: \"c8c74de0-5c52-4e2e-9f03-3b982a4e7667\") " pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496838    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-ovn\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496859    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-k8s-cni-cncf-io\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496856    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cni-binary-copy\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496891    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-lib-modules\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496904    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-multus\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496932    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-host\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496937    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-node-log\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.496921 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.496938    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-var-lib-kubelet\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.497014    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-log-socket\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.497031    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-etc-kubernetes\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.497040    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-socket-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.497075    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-sys-fs\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.497102    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-var-lib-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.497122    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-kubelet\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.497209    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.497273    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret podName:0d303f31-40f2-46fc-9bb1-73ba2851a5ec nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.997254861 +0000 UTC m=+3.134860474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret") pod "global-pull-secret-syncer-ddlpl" (UID: "0d303f31-40f2-46fc-9bb1-73ba2851a5ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.497320    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-systemd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.497359    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-system-cni-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.497556 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.497362    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.498342 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.498320    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-tuned\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.498434 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.498411    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-tmp\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.498494 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.498448    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-iptables-alerter-script\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.524822 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.524786    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4tt\" (UniqueName: \"kubernetes.io/projected/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-kube-api-access-9m4tt\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.527709 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.527635    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5txc\" (UniqueName: \"kubernetes.io/projected/1db5896f-bc1f-4965-9282-a344f5ad8e03-kube-api-access-b5txc\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.530335 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.530266    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbnn8\" (UniqueName: \"kubernetes.io/projected/a1df94c2-8be5-42df-9d04-7569e45f7648-kube-api-access-kbnn8\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.530664 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.530635    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr48h\" (UniqueName: \"kubernetes.io/projected/2a73c32c-6071-4eb7-8215-171cb490ee56-kube-api-access-xr48h\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:54:58.534131 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.534108    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:58.534131 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.534136    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:58.534322 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.534148    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lkz58 for pod openshift-network-diagnostics/network-check-target-rjwmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:58.534322 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:58.534212    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58 podName:23330a95-b468-4fa4-84d7-d5d58fbdf1cb nodeName:}" failed. No retries permitted until 2026-10-30 02:54:59.034193849 +0000 UTC m=+3.171799454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lkz58" (UniqueName: "kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58") pod "network-check-target-rjwmx" (UID: "23330a95-b468-4fa4-84d7-d5d58fbdf1cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:58.549606 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.549566    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b86p\" (UniqueName: \"kubernetes.io/projected/27811319-29a6-459b-a1e3-8325c6cc2274-kube-api-access-2b86p\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.552888 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.552855    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nflm\" (UniqueName: \"kubernetes.io/projected/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-kube-api-access-4nflm\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.553655 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.553613    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfjxv\" (UniqueName: \"kubernetes.io/projected/ac850265-e643-4365-b318-3507b8c838b9-kube-api-access-tfjxv\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.557375 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.557350    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjv7m\" (UniqueName: \"kubernetes.io/projected/603f9058-eea9-4899-872c-503c7111dcc7-kube-api-access-tjv7m\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.559038 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.558973    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2hwt\" (UniqueName: \"kubernetes.io/projected/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-kube-api-access-d2hwt\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.598695 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.598660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:54:58.609298 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.609255    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:54:58.617681 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:58.617616    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca98f80_def1_4e19_8bc6_0f8327e2a5f1.slice/crio-b8fd7b271a8f40ef7dbe3a14e0b09cd73a6ad62f6722a3b7f2ad86e13aa5ec10 WatchSource:0}: Error finding container b8fd7b271a8f40ef7dbe3a14e0b09cd73a6ad62f6722a3b7f2ad86e13aa5ec10: Status 404 returned error can't find the container with id b8fd7b271a8f40ef7dbe3a14e0b09cd73a6ad62f6722a3b7f2ad86e13aa5ec10
Oct 30 02:54:58.620178 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:58.620135    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c74de0_5c52_4e2e_9f03_3b982a4e7667.slice/crio-e956f9e234a728c0914d1dea985a699d79c229efa2ecdb6f4458e3aadf6a4fcd WatchSource:0}: Error finding container e956f9e234a728c0914d1dea985a699d79c229efa2ecdb6f4458e3aadf6a4fcd: Status 404 returned error can't find the container with id e956f9e234a728c0914d1dea985a699d79c229efa2ecdb6f4458e3aadf6a4fcd
Oct 30 02:54:58.627451 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.627424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:54:58.631475 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.631448    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g5vjw"
Oct 30 02:54:58.639809 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:58.639776    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1df94c2_8be5_42df_9d04_7569e45f7648.slice/crio-25da7a4ec4f13430449973faad5114f078e866c970a54b5c43bfb402da790236 WatchSource:0}: Error finding container 25da7a4ec4f13430449973faad5114f078e866c970a54b5c43bfb402da790236: Status 404 returned error can't find the container with id 25da7a4ec4f13430449973faad5114f078e866c970a54b5c43bfb402da790236
Oct 30 02:54:58.645859 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.645831    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:54:58.669921 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:58.669865    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cb55cf_0333_4d56_a6e0_4c13d0bd70a5.slice/crio-5e173113865d2ad09afa50c01803a24dd78262de029f3d578c38abdf0c300a3a WatchSource:0}: Error finding container 5e173113865d2ad09afa50c01803a24dd78262de029f3d578c38abdf0c300a3a: Status 404 returned error can't find the container with id 5e173113865d2ad09afa50c01803a24dd78262de029f3d578c38abdf0c300a3a
Oct 30 02:54:58.671020 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.670998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:54:58.675684 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.675658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:54:58.686269 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.686240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:54:58.686585 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:58.686556    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33044703_7fec_4a2f_a1c7_cde0e8dd50bc.slice/crio-7d57fb178fd29aecbb79ee7c69db7d63e5010270fe678c2a5607032b0618c730 WatchSource:0}: Error finding container 7d57fb178fd29aecbb79ee7c69db7d63e5010270fe678c2a5607032b0618c730: Status 404 returned error can't find the container with id 7d57fb178fd29aecbb79ee7c69db7d63e5010270fe678c2a5607032b0618c730
Oct 30 02:54:58.687789 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:58.687761    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27811319_29a6_459b_a1e3_8325c6cc2274.slice/crio-0f9167606fbccd6136d34ef05ec6dc82a0dbf080e7834191c193a2f0c39dc9fd WatchSource:0}: Error finding container 0f9167606fbccd6136d34ef05ec6dc82a0dbf080e7834191c193a2f0c39dc9fd: Status 404 returned error can't find the container with id 0f9167606fbccd6136d34ef05ec6dc82a0dbf080e7834191c193a2f0c39dc9fd
Oct 30 02:54:58.692628 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:58.692433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:54:58.701453 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:58.701423    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db5896f_bc1f_4965_9282_a344f5ad8e03.slice/crio-5c2f2a08082355357dd3ca1e9860dcf6351e8a96d53842bd82f06822a86ee9b2 WatchSource:0}: Error finding container 5c2f2a08082355357dd3ca1e9860dcf6351e8a96d53842bd82f06822a86ee9b2: Status 404 returned error can't find the container with id 5c2f2a08082355357dd3ca1e9860dcf6351e8a96d53842bd82f06822a86ee9b2
Oct 30 02:54:58.705675 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:54:58.705650    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603f9058_eea9_4899_872c_503c7111dcc7.slice/crio-3715466adbb270ef72769304c6f8bbad7bb59bd00ae8126b6cc0858d41a46132 WatchSource:0}: Error finding container 3715466adbb270ef72769304c6f8bbad7bb59bd00ae8126b6cc0858d41a46132: Status 404 returned error can't find the container with id 3715466adbb270ef72769304c6f8bbad7bb59bd00ae8126b6cc0858d41a46132
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.092716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.092810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.092852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:59.092932    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:59.092986    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:59.093024    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs podName:2a73c32c-6071-4eb7-8215-171cb490ee56 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:00.09299849 +0000 UTC m=+4.230604102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs") pod "network-metrics-daemon-ndbtf" (UID: "2a73c32c-6071-4eb7-8215-171cb490ee56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:59.093042    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:59.093047    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret podName:0d303f31-40f2-46fc-9bb1-73ba2851a5ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:00.093036315 +0000 UTC m=+4.230641910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret") pod "global-pull-secret-syncer-ddlpl" (UID: "0d303f31-40f2-46fc-9bb1-73ba2851a5ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:59.093067    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:59.093114    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lkz58 for pod openshift-network-diagnostics/network-check-target-rjwmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:59.093416 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:54:59.093273    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58 podName:23330a95-b468-4fa4-84d7-d5d58fbdf1cb nodeName:}" failed. No retries permitted until 2026-10-30 02:55:00.093151634 +0000 UTC m=+4.230757257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lkz58" (UniqueName: "kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58") pod "network-check-target-rjwmx" (UID: "23330a95-b468-4fa4-84d7-d5d58fbdf1cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:59.335232 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.335151    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:58 +0000 UTC" deadline="2028-04-12 22:31:29.188824266 +0000 UTC"
Oct 30 02:54:59.335232 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.335233    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12739h36m29.853597503s"
Oct 30 02:54:59.384019 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.383945    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9" event={"ID":"27811319-29a6-459b-a1e3-8325c6cc2274","Type":"ContainerStarted","Data":"0f9167606fbccd6136d34ef05ec6dc82a0dbf080e7834191c193a2f0c39dc9fd"}
Oct 30 02:54:59.399531 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.399482    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vsx5v" event={"ID":"ac850265-e643-4365-b318-3507b8c838b9","Type":"ContainerStarted","Data":"5e14ae5ecfb06b378739119d38b03970e4b299c88e268634c12b0d6974ead0f8"}
Oct 30 02:54:59.403629 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.403588    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g5vjw" event={"ID":"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5","Type":"ContainerStarted","Data":"5e173113865d2ad09afa50c01803a24dd78262de029f3d578c38abdf0c300a3a"}
Oct 30 02:54:59.407737 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.407696    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m2tdr" event={"ID":"c8c74de0-5c52-4e2e-9f03-3b982a4e7667","Type":"ContainerStarted","Data":"e956f9e234a728c0914d1dea985a699d79c229efa2ecdb6f4458e3aadf6a4fcd"}
Oct 30 02:54:59.412615 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.412573    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6j76z" event={"ID":"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1","Type":"ContainerStarted","Data":"b8fd7b271a8f40ef7dbe3a14e0b09cd73a6ad62f6722a3b7f2ad86e13aa5ec10"}
Oct 30 02:54:59.429304 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.429261    2580 generic.go:356] "Generic (PLEG): container finished" podID="2464e4e1bac4126b7f6dfa935cfd24ca" containerID="6264fc73ab0dde912ae68edc2f30144e219c05ba82303cc30d25c11354ff5c95" exitCode=0
Oct 30 02:54:59.429509 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.429398    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal" event={"ID":"2464e4e1bac4126b7f6dfa935cfd24ca","Type":"ContainerDied","Data":"6264fc73ab0dde912ae68edc2f30144e219c05ba82303cc30d25c11354ff5c95"}
Oct 30 02:54:59.454846 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.454798    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p6dt8" event={"ID":"1db5896f-bc1f-4965-9282-a344f5ad8e03","Type":"ContainerStarted","Data":"5c2f2a08082355357dd3ca1e9860dcf6351e8a96d53842bd82f06822a86ee9b2"}
Oct 30 02:54:59.464180 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.464115    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f8mgk" event={"ID":"33044703-7fec-4a2f-a1c7-cde0e8dd50bc","Type":"ContainerStarted","Data":"7d57fb178fd29aecbb79ee7c69db7d63e5010270fe678c2a5607032b0618c730"}
Oct 30 02:54:59.471147 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.471028    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t" event={"ID":"a1df94c2-8be5-42df-9d04-7569e45f7648","Type":"ContainerStarted","Data":"25da7a4ec4f13430449973faad5114f078e866c970a54b5c43bfb402da790236"}
Oct 30 02:54:59.483471 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.482939    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal" event={"ID":"868aae1459799721b266e5e297e80105","Type":"ContainerStarted","Data":"235f54fc267f63fec7acd59a6bcc5bf4007bb3fe32ed2ee6eda0803643a7006e"}
Oct 30 02:54:59.497187 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:54:59.497146    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cspkf" event={"ID":"603f9058-eea9-4899-872c-503c7111dcc7","Type":"ContainerStarted","Data":"3715466adbb270ef72769304c6f8bbad7bb59bd00ae8126b6cc0858d41a46132"}
Oct 30 02:55:00.104396 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.104289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:00.104396 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.104370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:00.104675 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.104420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:00.104675 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.104583    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:00.104675 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.104654    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs podName:2a73c32c-6071-4eb7-8215-171cb490ee56 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:02.104634626 +0000 UTC m=+6.242240233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs") pod "network-metrics-daemon-ndbtf" (UID: "2a73c32c-6071-4eb7-8215-171cb490ee56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:00.105163 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.105141    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:55:00.105273 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.105169    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:55:00.105273 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.105182    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lkz58 for pod openshift-network-diagnostics/network-check-target-rjwmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:00.105273 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.105244    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58 podName:23330a95-b468-4fa4-84d7-d5d58fbdf1cb nodeName:}" failed. No retries permitted until 2026-10-30 02:55:02.105227734 +0000 UTC m=+6.242833340 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lkz58" (UniqueName: "kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58") pod "network-check-target-rjwmx" (UID: "23330a95-b468-4fa4-84d7-d5d58fbdf1cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:00.105468 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.105312    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:00.105468 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.105346    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret podName:0d303f31-40f2-46fc-9bb1-73ba2851a5ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:02.105334666 +0000 UTC m=+6.242940272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret") pod "global-pull-secret-syncer-ddlpl" (UID: "0d303f31-40f2-46fc-9bb1-73ba2851a5ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:00.336549 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.336423    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:58 +0000 UTC" deadline="2028-07-01 08:28:05.343995168 +0000 UTC"
Oct 30 02:55:00.336549 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.336495    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14645h33m5.007505283s"
Oct 30 02:55:00.353820 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.353173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:00.353820 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.353245    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:00.353820 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.353238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:00.353820 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.353376    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:00.353820 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.353497    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:00.353820 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:00.353623    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:00.539850 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.539750    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal" event={"ID":"2464e4e1bac4126b7f6dfa935cfd24ca","Type":"ContainerStarted","Data":"bc4115256dcc06419bf50ce7826ff76abeb44ca935181e9b15f7df316e1b0e71"}
Oct 30 02:55:00.587268 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.587191    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal" podStartSLOduration=3.587165003 podStartE2EDuration="3.587165003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:59.528560329 +0000 UTC m=+3.666165944" watchObservedRunningTime="2026-10-30 02:55:00.587165003 +0000 UTC m=+4.724770629"
Oct 30 02:55:00.588336 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:00.587895    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal" podStartSLOduration=3.587876414 podStartE2EDuration="3.587876414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:00.586551634 +0000 UTC m=+4.724157247" watchObservedRunningTime="2026-10-30 02:55:00.587876414 +0000 UTC m=+4.725482031"
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:02.121831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:02.121918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:02.121968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.122108    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.122179    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs podName:2a73c32c-6071-4eb7-8215-171cb490ee56 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:06.122160212 +0000 UTC m=+10.259765828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs") pod "network-metrics-daemon-ndbtf" (UID: "2a73c32c-6071-4eb7-8215-171cb490ee56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.122512    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.122528    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.122537    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lkz58 for pod openshift-network-diagnostics/network-check-target-rjwmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.122582    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58 podName:23330a95-b468-4fa4-84d7-d5d58fbdf1cb nodeName:}" failed. No retries permitted until 2026-10-30 02:55:06.122567761 +0000 UTC m=+10.260173368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lkz58" (UniqueName: "kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58") pod "network-check-target-rjwmx" (UID: "23330a95-b468-4fa4-84d7-d5d58fbdf1cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.122645    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:02.122819 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.122674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret podName:0d303f31-40f2-46fc-9bb1-73ba2851a5ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:06.122667035 +0000 UTC m=+10.260272638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret") pod "global-pull-secret-syncer-ddlpl" (UID: "0d303f31-40f2-46fc-9bb1-73ba2851a5ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:02.353593 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:02.353042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:02.353593 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:02.353181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:02.353593 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.353198    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:02.353593 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:02.353223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:02.353593 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.353327    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:02.353984 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:02.353912    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:02.566357 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:02.566308    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m2tdr" event={"ID":"c8c74de0-5c52-4e2e-9f03-3b982a4e7667","Type":"ContainerStarted","Data":"750d986e9dc42595c191f1abeb8d2425b1e82b8043f7a9b43885496c0dce8fa5"}
Oct 30 02:55:02.578122 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:02.577994    2580 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="f005b28ce8da21e606b0f3c721b4d4bc3a8ecdfd4efe0810e591715676eacbb8" exitCode=0
Oct 30 02:55:02.578122 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:02.578055    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cspkf" event={"ID":"603f9058-eea9-4899-872c-503c7111dcc7","Type":"ContainerDied","Data":"f005b28ce8da21e606b0f3c721b4d4bc3a8ecdfd4efe0810e591715676eacbb8"}
Oct 30 02:55:03.624597 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:03.624519    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m2tdr" podStartSLOduration=3.936535488 podStartE2EDuration="7.624494155s" totalImagesPullingTime="3.687958667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:03.623555734 +0000 UTC m=+7.761161352" watchObservedRunningTime="2026-10-30 02:55:03.624494155 +0000 UTC m=+7.762099789"
Oct 30 02:55:04.352923 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:04.352050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:04.352923 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:04.352189    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:04.352923 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:04.352605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:04.352923 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:04.352706    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:04.352923 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:04.352786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:04.352923 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:04.352870    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:04.589938 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:04.589356    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f8mgk" event={"ID":"33044703-7fec-4a2f-a1c7-cde0e8dd50bc","Type":"ContainerStarted","Data":"886692d9e6df25584546b37cca1f0169ee1951ce229246c62a3f4a9e177ac39e"}
Oct 30 02:55:04.592149 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:04.592118    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t" event={"ID":"a1df94c2-8be5-42df-9d04-7569e45f7648","Type":"ContainerStarted","Data":"1e63466cb9689189c29e1cd4d6e4e83c7f2ff01b384fd79444624b67a951345a"}
Oct 30 02:55:05.974446 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:05.974413    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:55:06.163256 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:06.162883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:06.163256 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:06.162996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:06.163256 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.163027    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:06.163256 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:06.163034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:06.163256 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.163095    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs podName:2a73c32c-6071-4eb7-8215-171cb490ee56 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:14.163077444 +0000 UTC m=+18.300683038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs") pod "network-metrics-daemon-ndbtf" (UID: "2a73c32c-6071-4eb7-8215-171cb490ee56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:06.163256 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.163128    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:06.163256 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.163161    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret podName:0d303f31-40f2-46fc-9bb1-73ba2851a5ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:14.163151295 +0000 UTC m=+18.300756898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret") pod "global-pull-secret-syncer-ddlpl" (UID: "0d303f31-40f2-46fc-9bb1-73ba2851a5ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:06.163256 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.163184    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:55:06.163256 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.163216    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:55:06.163256 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.163230    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lkz58 for pod openshift-network-diagnostics/network-check-target-rjwmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:06.163846 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.163281    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58 podName:23330a95-b468-4fa4-84d7-d5d58fbdf1cb nodeName:}" failed. No retries permitted until 2026-10-30 02:55:14.163265062 +0000 UTC m=+18.300870657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lkz58" (UniqueName: "kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58") pod "network-check-target-rjwmx" (UID: "23330a95-b468-4fa4-84d7-d5d58fbdf1cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:06.354783 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:06.354740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:06.354983 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.354881    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:06.355357 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:06.355331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:06.356430 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.356366    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:06.361931 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:06.359175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:06.368315 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:06.362588    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:55:05.974438453Z","UUID":"27d8eef2-1b03-4bd0-b49f-ef4e48e32dee","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:55:06.369038 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:06.368991    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:06.371126 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:06.371101    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:55:06.371254 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:06.371145    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:55:06.606186 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:06.606077    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t" event={"ID":"a1df94c2-8be5-42df-9d04-7569e45f7648","Type":"ContainerStarted","Data":"4d0a32a602591387e549cf30308e449192dfa6d1018fabd7fa7be57c5cd0cd52"}
Oct 30 02:55:07.360592 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:07.360357    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:55:07.361410 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:07.361386    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:55:07.402877 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:07.402764    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f8mgk" podStartSLOduration=6.297791869 podStartE2EDuration="11.402744832s" totalImagesPullingTime="5.104952963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:04.632567854 +0000 UTC m=+8.770173466" watchObservedRunningTime="2026-10-30 02:55:07.402744832 +0000 UTC m=+11.540350448"
Oct 30 02:55:07.612016 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:07.611977    2580 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="16753f30a8904d7eaad1485a0615f62e2d76b4f10970a116cebb3f4c9830d221" exitCode=0
Oct 30 02:55:07.612179 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:07.612072    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cspkf" event={"ID":"603f9058-eea9-4899-872c-503c7111dcc7","Type":"ContainerDied","Data":"16753f30a8904d7eaad1485a0615f62e2d76b4f10970a116cebb3f4c9830d221"}
Oct 30 02:55:07.616827 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:07.616794    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vsx5v" event={"ID":"ac850265-e643-4365-b318-3507b8c838b9","Type":"ContainerStarted","Data":"c2b368a344939688741d9706f052c85fa6a1f3cd0bd5cfb2967aa94fcd84cba7"}
Oct 30 02:55:07.619248 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:07.619218    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g5vjw" event={"ID":"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5","Type":"ContainerStarted","Data":"37cc1b37ab2d52cecf0239ee12acca8ad550fedbd9156b5af1fd656b8691c6c5"}
Oct 30 02:55:07.706172 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:07.706108    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-g5vjw" podStartSLOduration=3.8402592760000003 podStartE2EDuration="11.706088616s" totalImagesPullingTime="7.86582934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:07.705388269 +0000 UTC m=+11.842993904" watchObservedRunningTime="2026-10-30 02:55:07.706088616 +0000 UTC m=+11.843694228"
Oct 30 02:55:07.748137 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:07.747620    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vsx5v" podStartSLOduration=3.7404010210000003 podStartE2EDuration="11.747604879s" totalImagesPullingTime="8.007203858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:07.747221081 +0000 UTC m=+11.884826694" watchObservedRunningTime="2026-10-30 02:55:07.747604879 +0000 UTC m=+11.885210491"
Oct 30 02:55:08.354671 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:08.354640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:08.354807 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:08.354773    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:08.355217 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:08.355195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:08.355333 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:08.355313    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:08.355402 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:08.355381    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:08.355467 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:08.355449    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:08.623440 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:08.623343    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6j76z" event={"ID":"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1","Type":"ContainerStarted","Data":"18cc6f850ef43119878e173447769cc44bef461d56cc5b8733d1edbebddaf4e2"}
Oct 30 02:55:08.626707 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:08.625970    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t" event={"ID":"a1df94c2-8be5-42df-9d04-7569e45f7648","Type":"ContainerStarted","Data":"ff8db9f7fe513a5b6b7040b53b329b36448eb2fdc949ef92217fd4e4e22de92d"}
Oct 30 02:55:08.664625 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:08.664576    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6j76z" podStartSLOduration=4.408472343 podStartE2EDuration="12.664562242s" totalImagesPullingTime="8.256089899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:08.664304629 +0000 UTC m=+12.801910241" watchObservedRunningTime="2026-10-30 02:55:08.664562242 +0000 UTC m=+12.802167852"
Oct 30 02:55:09.630477 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:09.630441    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p6dt8" event={"ID":"1db5896f-bc1f-4965-9282-a344f5ad8e03","Type":"ContainerStarted","Data":"a80a2ee637241233a949e168befeed78018f2981ca64dfb22f4f31ffb24a3cd7"}
Oct 30 02:55:09.632838 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:09.632800    2580 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="5b9cefec968f60260836eb0576e0f29e4e28ba75b44d8430e4c6a7373d5d8845" exitCode=0
Oct 30 02:55:09.633005 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:09.632858    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cspkf" event={"ID":"603f9058-eea9-4899-872c-503c7111dcc7","Type":"ContainerDied","Data":"5b9cefec968f60260836eb0576e0f29e4e28ba75b44d8430e4c6a7373d5d8845"}
Oct 30 02:55:09.675643 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:09.675591    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t" podStartSLOduration=4.520028141 podStartE2EDuration="13.675571405s" totalImagesPullingTime="9.155543264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:08.711396821 +0000 UTC m=+12.849002434" watchObservedRunningTime="2026-10-30 02:55:09.675571405 +0000 UTC m=+13.813177019"
Oct 30 02:55:09.676076 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:09.676039    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p6dt8" podStartSLOduration=3.3393479 podStartE2EDuration="13.676031797s" totalImagesPullingTime="10.336683897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:09.67531274 +0000 UTC m=+13.812918352" watchObservedRunningTime="2026-10-30 02:55:09.676031797 +0000 UTC m=+13.813637439"
Oct 30 02:55:10.355317 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:10.355288    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:10.355542 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:10.355391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:10.355542 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:10.355501    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:10.355542 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:10.355519    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:10.355686 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:10.355579    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:10.355686 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:10.355645    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:11.641589 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:11.641369    2580 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="bcec0d38b085829af64dd83e30eff9a049b7e373400035b4d0f19059a88e92c5" exitCode=0
Oct 30 02:55:11.641589 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:11.641426    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cspkf" event={"ID":"603f9058-eea9-4899-872c-503c7111dcc7","Type":"ContainerDied","Data":"bcec0d38b085829af64dd83e30eff9a049b7e373400035b4d0f19059a88e92c5"}
Oct 30 02:55:12.352496 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:12.352463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:12.352673 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:12.352463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:12.352673 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:12.352605    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:12.352673 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:12.352463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:12.352832 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:12.352669    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:12.352832 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:12.352712    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:13.648665 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:13.648631    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cspkf" event={"ID":"603f9058-eea9-4899-872c-503c7111dcc7","Type":"ContainerStarted","Data":"aa20bb1117040ce4ff2876bde33cadbe24a7a2b50c0eca7c82aea19237e498cb"}
Oct 30 02:55:13.650466 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:13.650442    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9" event={"ID":"27811319-29a6-459b-a1e3-8325c6cc2274","Type":"ContainerStarted","Data":"1367dd4a8a269058d9438c9922ecb8c52934cd35801722fd8c37ca298f4661e1"}
Oct 30 02:55:13.650575 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:13.650506    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9" event={"ID":"27811319-29a6-459b-a1e3-8325c6cc2274","Type":"ContainerStarted","Data":"8fd14c24a60613a375b6fa79298a5c99c88d35b2d0d44f07738d1c9169c79798"}
Oct 30 02:55:14.226154 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.225964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:14.226296 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.226163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:14.226296 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.226189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:14.226296 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.226128    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:55:14.226296 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.226229    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:55:14.226296 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.226240    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lkz58 for pod openshift-network-diagnostics/network-check-target-rjwmx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:14.226296 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.226292    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58 podName:23330a95-b468-4fa4-84d7-d5d58fbdf1cb nodeName:}" failed. No retries permitted until 2026-10-30 02:55:30.226278408 +0000 UTC m=+34.363884018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lkz58" (UniqueName: "kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58") pod "network-check-target-rjwmx" (UID: "23330a95-b468-4fa4-84d7-d5d58fbdf1cb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:55:14.226296 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.226291    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:14.226525 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.226301    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:14.226525 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.226341    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs podName:2a73c32c-6071-4eb7-8215-171cb490ee56 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:30.226325398 +0000 UTC m=+34.363930989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs") pod "network-metrics-daemon-ndbtf" (UID: "2a73c32c-6071-4eb7-8215-171cb490ee56") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:55:14.226525 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.226354    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret podName:0d303f31-40f2-46fc-9bb1-73ba2851a5ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:30.226347988 +0000 UTC m=+34.363953577 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret") pod "global-pull-secret-syncer-ddlpl" (UID: "0d303f31-40f2-46fc-9bb1-73ba2851a5ec") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:55:14.351893 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.351852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:14.352070 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.351864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:14.352070 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.351992    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:14.352070 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.352009    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:14.352255 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.352121    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:14.352255 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:14.352206    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:14.654759 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.654669    2580 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="aa20bb1117040ce4ff2876bde33cadbe24a7a2b50c0eca7c82aea19237e498cb" exitCode=0
Oct 30 02:55:14.655241 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.654753    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cspkf" event={"ID":"603f9058-eea9-4899-872c-503c7111dcc7","Type":"ContainerDied","Data":"aa20bb1117040ce4ff2876bde33cadbe24a7a2b50c0eca7c82aea19237e498cb"}
Oct 30 02:55:14.658027 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.658004    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9" event={"ID":"27811319-29a6-459b-a1e3-8325c6cc2274","Type":"ContainerStarted","Data":"d0c634c65ca64b9b8983df2c6c1b14022b4732f80b5745d07a43c250c3c0603b"}
Oct 30 02:55:14.658127 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.658066    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9" event={"ID":"27811319-29a6-459b-a1e3-8325c6cc2274","Type":"ContainerStarted","Data":"64a5d1a90c28cb098baee798b5af6be8c9fd9c3e75084341fb2ba0d3f56c7851"}
Oct 30 02:55:14.658127 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.658096    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9" event={"ID":"27811319-29a6-459b-a1e3-8325c6cc2274","Type":"ContainerStarted","Data":"96c3c4a094f1e37af2950335dc1304bd20172eea0efa281e26d9431a5888bd06"}
Oct 30 02:55:14.658127 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:14.658125    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9" event={"ID":"27811319-29a6-459b-a1e3-8325c6cc2274","Type":"ContainerStarted","Data":"9df0577b477ee5b5652ee2e63f3872d54e68639745e19ea09bf52fa6c7f359c7"}
Oct 30 02:55:15.663695 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:15.663655    2580 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="07bb3e394e9db5e19921cd49bc377d57a9bce18a77f3120721ed9b03deab9128" exitCode=0
Oct 30 02:55:15.664215 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:15.663715    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cspkf" event={"ID":"603f9058-eea9-4899-872c-503c7111dcc7","Type":"ContainerDied","Data":"07bb3e394e9db5e19921cd49bc377d57a9bce18a77f3120721ed9b03deab9128"}
Oct 30 02:55:15.676598 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:15.676564    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:55:15.676764 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:15.676747    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:15.677368 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:15.677344    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:55:16.352875 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:16.352836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:16.353057 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:16.352953    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:16.353057 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:16.352978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:16.353182 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:16.352986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:16.353182 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:16.353144    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:16.353182 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:16.353071    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:16.668888 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:16.668810    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cspkf" event={"ID":"603f9058-eea9-4899-872c-503c7111dcc7","Type":"ContainerStarted","Data":"83ad3b92e803f9d18637b2048ea18e387ab24514536e1646b644b4c6cc0dc259"}
Oct 30 02:55:16.671833 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:16.671809    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9" event={"ID":"27811319-29a6-459b-a1e3-8325c6cc2274","Type":"ContainerStarted","Data":"b58010841f58f8829d6e76ffeba55a0950431e91f24858562a4046205915238b"}
Oct 30 02:55:16.744778 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:16.744726    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cspkf" podStartSLOduration=9.621285798 podStartE2EDuration="20.744711719s" totalImagesPullingTime="11.123425921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:16.731291442 +0000 UTC m=+20.868897055" watchObservedRunningTime="2026-10-30 02:55:16.744711719 +0000 UTC m=+20.882317330"
Oct 30 02:55:18.352162 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:18.351958    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:18.352724 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:18.352021    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:18.352724 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:18.352260    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:18.352724 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:18.352310    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:18.352724 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:18.352028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:18.352724 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:18.352454    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:18.679513 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:18.679430    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9" event={"ID":"27811319-29a6-459b-a1e3-8325c6cc2274","Type":"ContainerStarted","Data":"d3adff6bb6be802cd4f682565d7692309ebf20d0c8d73035f9ed75efcb3903da"}
Oct 30 02:55:18.679744 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:18.679690    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:18.679834 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:18.679771    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:18.696033 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:18.696001    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:18.775832 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:18.775784    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9" podStartSLOduration=7.993776023 podStartE2EDuration="22.775770725s" totalImagesPullingTime="14.781994702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:18.761920868 +0000 UTC m=+22.899526480" watchObservedRunningTime="2026-10-30 02:55:18.775770725 +0000 UTC m=+22.913376337"
Oct 30 02:55:19.682272 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:19.682234    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:19.701261 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:19.701232    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:20.088312 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:20.088280    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ddlpl"]
Oct 30 02:55:20.088518 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:20.088407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:20.088518 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:20.088508    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:20.091916 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:20.091856    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rjwmx"]
Oct 30 02:55:20.092061 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:20.092045    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:20.092183 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:20.092164    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:20.092599 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:20.092570    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ndbtf"]
Oct 30 02:55:20.092715 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:20.092699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:20.092823 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:20.092800    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:21.352008 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:21.351961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:21.352452 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:21.352091    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:22.352115 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:22.352020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:22.352465 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:22.352044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:22.352465 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:22.352156    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:22.352465 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:22.352193    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:23.351852 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:23.351760    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:23.352009 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:23.351891    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:24.352071 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:24.352030    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:24.352071 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:24.352060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:24.352641 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:24.352141    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ddlpl" podUID="0d303f31-40f2-46fc-9bb1-73ba2851a5ec"
Oct 30 02:55:24.352641 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:24.352288    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjwmx" podUID="23330a95-b468-4fa4-84d7-d5d58fbdf1cb"
Oct 30 02:55:25.352019 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:25.351980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:25.352219 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:25.352099    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndbtf" podUID="2a73c32c-6071-4eb7-8215-171cb490ee56"
Oct 30 02:55:26.139556 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.139531    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeReady"
Oct 30 02:55:26.139698 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.139662    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 02:55:26.197319 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.197228    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2d7p7"]
Oct 30 02:55:26.199738 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.199713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:26.200339 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.200313    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zxwqk"]
Oct 30 02:55:26.202771 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.202748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.218781 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.218747    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xzg75\""
Oct 30 02:55:26.222246 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.222226    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:55:26.222476 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.222460    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:55:26.222669 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.222655    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:55:26.224670 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.224654    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:55:26.225872 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.225844    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2d7p7"]
Oct 30 02:55:26.233147 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.233116    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zxwqk"]
Oct 30 02:55:26.306734 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.306701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-cert\") pod \"ingress-canary-2d7p7\" (UID: \"2c7ca2bd-121d-4ef9-ae00-4a139497c91d\") " pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:26.306943 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.306750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0213f1-3092-4a53-ba3a-1cec86701275-config-volume\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.306943 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.306772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0213f1-3092-4a53-ba3a-1cec86701275-tmp-dir\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.306943 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.306829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxgw\" (UniqueName: \"kubernetes.io/projected/bf0213f1-3092-4a53-ba3a-1cec86701275-kube-api-access-ttxgw\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.307052 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.306994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.307052 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.307040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c69r7\" (UniqueName: \"kubernetes.io/projected/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-kube-api-access-c69r7\") pod \"ingress-canary-2d7p7\" (UID: \"2c7ca2bd-121d-4ef9-ae00-4a139497c91d\") " pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:26.319434 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.319415    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9wgn\""
Oct 30 02:55:26.319434 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.319422    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:55:26.354183 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.354139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:26.354640 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.354239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:26.371083 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.370276    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kdtjd\""
Oct 30 02:55:26.371083 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.370368    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:55:26.371083 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.370433    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bckpk\""
Oct 30 02:55:26.371083 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.370567    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:55:26.371083 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.370879    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:55:26.407324 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.407277    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.407538 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.407336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c69r7\" (UniqueName: \"kubernetes.io/projected/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-kube-api-access-c69r7\") pod \"ingress-canary-2d7p7\" (UID: \"2c7ca2bd-121d-4ef9-ae00-4a139497c91d\") " pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:26.407538 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.407357    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-cert\") pod \"ingress-canary-2d7p7\" (UID: \"2c7ca2bd-121d-4ef9-ae00-4a139497c91d\") " pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:26.407538 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.407384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0213f1-3092-4a53-ba3a-1cec86701275-config-volume\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.407538 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.407403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0213f1-3092-4a53-ba3a-1cec86701275-tmp-dir\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.407538 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:26.407416    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:26.407538 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.407427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxgw\" (UniqueName: \"kubernetes.io/projected/bf0213f1-3092-4a53-ba3a-1cec86701275-kube-api-access-ttxgw\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.407855 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:26.407644    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls podName:bf0213f1-3092-4a53-ba3a-1cec86701275 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:26.907620059 +0000 UTC m=+31.045225662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls") pod "dns-default-zxwqk" (UID: "bf0213f1-3092-4a53-ba3a-1cec86701275") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:26.407943 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.407855    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0213f1-3092-4a53-ba3a-1cec86701275-tmp-dir\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.409137 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.409108    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0213f1-3092-4a53-ba3a-1cec86701275-config-volume\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.411591 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.411564    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-cert\") pod \"ingress-canary-2d7p7\" (UID: \"2c7ca2bd-121d-4ef9-ae00-4a139497c91d\") " pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:26.433604 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.433566    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxgw\" (UniqueName: \"kubernetes.io/projected/bf0213f1-3092-4a53-ba3a-1cec86701275-kube-api-access-ttxgw\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.434185 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.434164    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c69r7\" (UniqueName: \"kubernetes.io/projected/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-kube-api-access-c69r7\") pod \"ingress-canary-2d7p7\" (UID: \"2c7ca2bd-121d-4ef9-ae00-4a139497c91d\") " pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:26.513289 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.513198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:26.527472 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.527445    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pgwbh"]
Oct 30 02:55:26.530963 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.530916    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.549550 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.549525    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:55:26.552326 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.552306    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:55:26.552446 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.552350    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:55:26.552531 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.552514    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:55:26.552746 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.552721    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d8b4w\""
Oct 30 02:55:26.553920 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.553880    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pgwbh"]
Oct 30 02:55:26.596656 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.595727    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-clqt4-925xq"]
Oct 30 02:55:26.598775 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.598745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.608530 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.608488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-crio-socket\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.608669 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.608541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.608669 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.608583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45bk\" (UniqueName: \"kubernetes.io/projected/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-api-access-v45bk\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.608669 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.608607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-data-volume\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.608669 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.608624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.618946 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.618340    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-clqt4-925xq"]
Oct 30 02:55:26.635402 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.635375    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f2mz6\""
Oct 30 02:55:26.650654 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.650624    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:55:26.665740 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.665710    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:55:26.709439 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-data-volume\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.709439 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.709671 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-crio-socket\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.709671 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.709671 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-trusted-ca-bundle\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.709671 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45bk\" (UniqueName: \"kubernetes.io/projected/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-api-access-v45bk\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.709671 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:26.709643    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:55:26.709888 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709652    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e18c7963-1218-4b83-837b-391b7d3edd22-archives-path\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.709888 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:26.709651    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:26.709888 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-service-ca-bundle\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.709888 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:26.709756    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:27.209738205 +0000 UTC m=+31.347343795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:26.709888 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709753    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-crio-socket\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.709888 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:26.709797    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:27.209770515 +0000 UTC m=+31.347376116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : configmap "kube-rbac-proxy" not found
Oct 30 02:55:26.709888 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709830    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-data-volume\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.710163 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.709921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8vxl\" (UniqueName: \"kubernetes.io/projected/e18c7963-1218-4b83-837b-391b7d3edd22-kube-api-access-p8vxl\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.717373 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.717346    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7"]
Oct 30 02:55:26.720263 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.720249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7"
Oct 30 02:55:26.736537 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.736510    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-chfb9\""
Oct 30 02:55:26.740878 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.740848    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7"]
Oct 30 02:55:26.740878 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.740866    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45bk\" (UniqueName: \"kubernetes.io/projected/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-api-access-v45bk\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:26.749746 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.749720    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:55:26.749886 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.749771    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:55:26.810766 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.810731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9cqv\" (UniqueName: \"kubernetes.io/projected/32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65-kube-api-access-j9cqv\") pod \"migrator-66df5d496b-ctxf7\" (UID: \"32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7"
Oct 30 02:55:26.810968 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.810785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e18c7963-1218-4b83-837b-391b7d3edd22-archives-path\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.810968 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.810803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-service-ca-bundle\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.810968 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.810885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8vxl\" (UniqueName: \"kubernetes.io/projected/e18c7963-1218-4b83-837b-391b7d3edd22-kube-api-access-p8vxl\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.811094 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.811045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-trusted-ca-bundle\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.811458 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.811437    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e18c7963-1218-4b83-837b-391b7d3edd22-archives-path\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.811509 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.811486    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-service-ca-bundle\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.811986 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.811969    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-trusted-ca-bundle\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.823835 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.823811    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2d7p7"]
Oct 30 02:55:26.826597 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:55:26.826569    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7ca2bd_121d_4ef9_ae00_4a139497c91d.slice/crio-49fabea07507cca26bdcf3a6577a1a06fa8fd93cf4235af5c4e726e14e5b0f13 WatchSource:0}: Error finding container 49fabea07507cca26bdcf3a6577a1a06fa8fd93cf4235af5c4e726e14e5b0f13: Status 404 returned error can't find the container with id 49fabea07507cca26bdcf3a6577a1a06fa8fd93cf4235af5c4e726e14e5b0f13
Oct 30 02:55:26.835631 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.835600    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8vxl\" (UniqueName: \"kubernetes.io/projected/e18c7963-1218-4b83-837b-391b7d3edd22-kube-api-access-p8vxl\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.911191 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.911155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:26.911475 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.911452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9cqv\" (UniqueName: \"kubernetes.io/projected/32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65-kube-api-access-j9cqv\") pod \"migrator-66df5d496b-ctxf7\" (UID: \"32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7"
Oct 30 02:55:26.911530 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.911490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:26.911600 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:26.911588    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:26.911646 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:26.911638    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls podName:bf0213f1-3092-4a53-ba3a-1cec86701275 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:27.911625469 +0000 UTC m=+32.049231062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls") pod "dns-default-zxwqk" (UID: "bf0213f1-3092-4a53-ba3a-1cec86701275") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:26.937192 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:26.937154    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9cqv\" (UniqueName: \"kubernetes.io/projected/32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65-kube-api-access-j9cqv\") pod \"migrator-66df5d496b-ctxf7\" (UID: \"32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7"
Oct 30 02:55:27.031892 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.031842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7"
Oct 30 02:55:27.214777 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.213794    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:27.214777 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.213888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:27.214777 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:27.213941    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:55:27.214777 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:27.214022    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:28.21399775 +0000 UTC m=+32.351603365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : configmap "kube-rbac-proxy" not found
Oct 30 02:55:27.214777 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:27.214102    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:27.214777 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:27.214182    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:28.214164817 +0000 UTC m=+32.351770407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:27.215060 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.214950    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-clqt4-925xq"]
Oct 30 02:55:27.216296 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:55:27.216267    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18c7963_1218_4b83_837b_391b7d3edd22.slice/crio-15f683ad8289c1a7eab25fbfddc39ff6ec5f73becd79c0e5e6bb7dc6ccc71690 WatchSource:0}: Error finding container 15f683ad8289c1a7eab25fbfddc39ff6ec5f73becd79c0e5e6bb7dc6ccc71690: Status 404 returned error can't find the container with id 15f683ad8289c1a7eab25fbfddc39ff6ec5f73becd79c0e5e6bb7dc6ccc71690
Oct 30 02:55:27.327401 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.327374    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7"]
Oct 30 02:55:27.330204 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:55:27.330139    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a2baa7_1ca5_4dd5_ba93_44e84e3e6f65.slice/crio-b50c6dc549d18e7f6dbca52525d6276f51119940d279437939a7abb125b5d057 WatchSource:0}: Error finding container b50c6dc549d18e7f6dbca52525d6276f51119940d279437939a7abb125b5d057: Status 404 returned error can't find the container with id b50c6dc549d18e7f6dbca52525d6276f51119940d279437939a7abb125b5d057
Oct 30 02:55:27.352384 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.352349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:27.368848 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.368812    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:55:27.368848 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.368820    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vzrlw\""
Oct 30 02:55:27.705338 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.705248    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7" event={"ID":"32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65","Type":"ContainerStarted","Data":"b50c6dc549d18e7f6dbca52525d6276f51119940d279437939a7abb125b5d057"}
Oct 30 02:55:27.707409 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.707331    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-clqt4-925xq" event={"ID":"e18c7963-1218-4b83-837b-391b7d3edd22","Type":"ContainerStarted","Data":"15f683ad8289c1a7eab25fbfddc39ff6ec5f73becd79c0e5e6bb7dc6ccc71690"}
Oct 30 02:55:27.708960 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.708918    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2d7p7" event={"ID":"2c7ca2bd-121d-4ef9-ae00-4a139497c91d","Type":"ContainerStarted","Data":"49fabea07507cca26bdcf3a6577a1a06fa8fd93cf4235af5c4e726e14e5b0f13"}
Oct 30 02:55:27.920416 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:27.920383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:27.920621 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:27.920520    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:27.920621 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:27.920576    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls podName:bf0213f1-3092-4a53-ba3a-1cec86701275 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:29.920562283 +0000 UTC m=+34.058167873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls") pod "dns-default-zxwqk" (UID: "bf0213f1-3092-4a53-ba3a-1cec86701275") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:28.223811 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:28.223755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:28.224004 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:28.223864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:28.224004 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:28.223998    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:55:28.224113 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:28.224051    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:30.224037422 +0000 UTC m=+34.361643023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : configmap "kube-rbac-proxy" not found
Oct 30 02:55:28.224162 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:28.224126    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:28.224162 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:28.224151    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:30.224143906 +0000 UTC m=+34.361749509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:29.715440 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:29.715226    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7" event={"ID":"32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65","Type":"ContainerStarted","Data":"52f217214c28d7d8d41b46edd0e6996a28bdcebd4b52d03bdb0edfd9843d8657"}
Oct 30 02:55:29.717194 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:29.717156    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-clqt4-925xq" event={"ID":"e18c7963-1218-4b83-837b-391b7d3edd22","Type":"ContainerStarted","Data":"88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc"}
Oct 30 02:55:29.719075 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:29.719043    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2d7p7" event={"ID":"2c7ca2bd-121d-4ef9-ae00-4a139497c91d","Type":"ContainerStarted","Data":"631706543fddd41022218bb74c898e114f99d0d5c18da4fa780d2921c8423238"}
Oct 30 02:55:29.758881 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:29.758728    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7" podStartSLOduration=2.390156079 podStartE2EDuration="3.758711951s" totalImagesPullingTime="1.368555872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:29.757949664 +0000 UTC m=+33.895555275" watchObservedRunningTime="2026-10-30 02:55:29.758711951 +0000 UTC m=+33.896317562"
Oct 30 02:55:29.816694 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:29.816645    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-clqt4-925xq" podStartSLOduration=1.808697736 podStartE2EDuration="3.816629975s" totalImagesPullingTime="2.007932239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:29.816210743 +0000 UTC m=+33.953816355" watchObservedRunningTime="2026-10-30 02:55:29.816629975 +0000 UTC m=+33.954235585"
Oct 30 02:55:29.859634 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:29.859564    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2d7p7" podStartSLOduration=1.189015797 podStartE2EDuration="3.859527529s" totalImagesPullingTime="2.670511732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:29.858804796 +0000 UTC m=+33.996410406" watchObservedRunningTime="2026-10-30 02:55:29.859527529 +0000 UTC m=+33.997133140"
Oct 30 02:55:29.941669 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:29.941628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:29.941853 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:29.941808    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:29.941982 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:29.941921    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls podName:bf0213f1-3092-4a53-ba3a-1cec86701275 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:33.941874703 +0000 UTC m=+38.079480305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls") pod "dns-default-zxwqk" (UID: "bf0213f1-3092-4a53-ba3a-1cec86701275") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:30.244134 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.244091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:30.244134 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.244132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:30.244378 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.244156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:30.244378 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.244185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:30.244378 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.244201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:30.244378 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:30.244259    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:55:30.244378 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:30.244275    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:55:30.244378 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:30.244315    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:34.244294786 +0000 UTC m=+38.381900399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : configmap "kube-rbac-proxy" not found
Oct 30 02:55:30.244378 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:30.244338    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs podName:2a73c32c-6071-4eb7-8215-171cb490ee56 nodeName:}" failed. No retries permitted until 2026-10-30 02:56:02.244322712 +0000 UTC m=+66.381928314 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs") pod "network-metrics-daemon-ndbtf" (UID: "2a73c32c-6071-4eb7-8215-171cb490ee56") : secret "metrics-daemon-secret" not found
Oct 30 02:55:30.244378 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:30.244275    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:30.244621 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:30.244421    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:34.244404576 +0000 UTC m=+38.382010174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:30.246825 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.246787    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:30.247226 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.247209    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:30.268334 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.268303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:30.289254 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.289228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:30.574602 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.574571    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ddlpl"]
Oct 30 02:55:30.575630 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:55:30.575602    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d303f31_40f2_46fc_9bb1_73ba2851a5ec.slice/crio-504a98366245e55532fe800e409e5e32641a6195e9f8edcef0453c44eefd4c15 WatchSource:0}: Error finding container 504a98366245e55532fe800e409e5e32641a6195e9f8edcef0453c44eefd4c15: Status 404 returned error can't find the container with id 504a98366245e55532fe800e409e5e32641a6195e9f8edcef0453c44eefd4c15
Oct 30 02:55:30.633793 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.633729    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rjwmx"]
Oct 30 02:55:30.635733 ip-10-0-130-130 kubenswrapper[2580]: W1030 02:55:30.635707    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23330a95_b468_4fa4_84d7_d5d58fbdf1cb.slice/crio-a8395834fd92499117320b142cc41fccfb4e8caf90bf4c3328194e68a1cef2a5 WatchSource:0}: Error finding container a8395834fd92499117320b142cc41fccfb4e8caf90bf4c3328194e68a1cef2a5: Status 404 returned error can't find the container with id a8395834fd92499117320b142cc41fccfb4e8caf90bf4c3328194e68a1cef2a5
Oct 30 02:55:30.722449 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.722407    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ddlpl" event={"ID":"0d303f31-40f2-46fc-9bb1-73ba2851a5ec","Type":"ContainerStarted","Data":"504a98366245e55532fe800e409e5e32641a6195e9f8edcef0453c44eefd4c15"}
Oct 30 02:55:30.723580 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:30.723553    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rjwmx" event={"ID":"23330a95-b468-4fa4-84d7-d5d58fbdf1cb","Type":"ContainerStarted","Data":"a8395834fd92499117320b142cc41fccfb4e8caf90bf4c3328194e68a1cef2a5"}
Oct 30 02:55:33.983240 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:33.983205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:33.983700 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:33.983376    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:33.983700 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:33.983455    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls podName:bf0213f1-3092-4a53-ba3a-1cec86701275 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:41.983435691 +0000 UTC m=+46.121041282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls") pod "dns-default-zxwqk" (UID: "bf0213f1-3092-4a53-ba3a-1cec86701275") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:34.286352 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:34.286312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:34.286502 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:34.286379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:34.286502 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:34.286484    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:34.286578 ip-10-0-130-130 kubenswrapper[2580]: E1030 02:55:34.286560    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:42.286540827 +0000 UTC m=+46.424146418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:34.286952 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:34.286934    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:34.738290 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:34.738196    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rjwmx" event={"ID":"23330a95-b468-4fa4-84d7-d5d58fbdf1cb","Type":"ContainerStarted","Data":"861bc330456b60c4dfb51a8990a6db90b037076b58328a89915c02b6b0ee1030"}
Oct 30 02:55:34.738456 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:34.738440    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:34.780337 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:34.780278    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rjwmx" podStartSLOduration=35.484744455 podStartE2EDuration="38.780260156s" totalImagesPullingTime="3.295515701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:34.77978778 +0000 UTC m=+38.917393417" watchObservedRunningTime="2026-10-30 02:55:34.780260156 +0000 UTC m=+38.917865768"
Oct 30 02:55:35.093056 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:35.093030    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vsx5v_ac850265-e643-4365-b318-3507b8c838b9/dns-node-resolver/0.log"
Oct 30 02:55:35.734820 ip-10-0-130-130 kubenswrapper[2580]: I1030 02:55:35.734698    2580 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:55:35.734820 ip-10-0-130-130 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 02:55:35.745343 ip-10-0-130-130 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 02:55:35.745514 ip-10-0-130-130 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 02:55:35.745564 ip-10-0-130-130 systemd[1]: kubelet.service: Consumed 2.508s CPU time.
Oct 30 02:55:35.755420 ip-10-0-130-130 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:55:35.843301 ip-10-0-130-130 kubenswrapper[4494]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:55:35.843301 ip-10-0-130-130 kubenswrapper[4494]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:55:35.843301 ip-10-0-130-130 kubenswrapper[4494]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:55:35.843301 ip-10-0-130-130 kubenswrapper[4494]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:55:35.845505 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845412    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:35.845505 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845506    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845510    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845513    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845516    4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845519    4494 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845522    4494 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845524    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845527    4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845529    4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845532    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845535    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845538    4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845541    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845544    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845547    4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845549    4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845552    4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845554    4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845557    4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:35.845577 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845559    4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845563    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845565    4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845568    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845570    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845573    4494 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845577    4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845580    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845583    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845586    4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845589    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845592    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845594    4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845597    4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845599    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845602    4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845605    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845607    4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845610    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:35.846077 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845612    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845614    4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845617    4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845620    4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845623    4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845627    4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845629    4494 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845632    4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845634    4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845636    4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845639    4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845642    4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845644    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845647    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845650    4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845653    4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845655    4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845657    4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845660    4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845664    4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:35.846555 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845666    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845670    4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845673    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845676    4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845678    4494 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845681    4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845683    4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845686    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845689    4494 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845691    4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845693    4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845696    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845698    4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845701    4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845703    4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845705    4494 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845708    4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845711    4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845713    4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845716    4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:35.847110 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845718    4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845721    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845724    4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845726    4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845729    4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845732    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845735    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845737    4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845741    4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845745    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845749    4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845752    4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845755    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845757    4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845760    4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845763    4494 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845765    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845767    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845770    4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:35.847618 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845772    4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845775    4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845780    4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845784    4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845787    4494 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845789    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845792    4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.845794    4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846626    4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846634    4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846639    4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846643    4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846647    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846650    4494 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846653    4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846655    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846658    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846660    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846663    4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:35.848105 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846666    4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846668    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846671    4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846674    4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846676    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846679    4494 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846682    4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846684    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846687    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846689    4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846692    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846695    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846697    4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846700    4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846702    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846705    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846707    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846710    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846712    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:35.848581 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846714    4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846717    4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846719    4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846721    4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846724    4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846727    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846730    4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846732    4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846735    4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846738    4494 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846740    4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846743    4494 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846745    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846748    4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846750    4494 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846752    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846755    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846757    4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846760    4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846763    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:35.849067 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846765    4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846768    4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846770    4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846772    4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846775    4494 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846777    4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846780    4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846782    4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846784    4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846787    4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846789    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846792    4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846794    4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846796    4494 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846800    4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846802    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846805    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846807    4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846810    4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846813    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:35.849594 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846816    4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846819    4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846821    4494 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846823    4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846826    4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846831    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846834    4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846836    4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846839    4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846841    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846844    4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846846    4494 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846849    4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846851    4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846854    4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846856    4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846859    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846873    4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846876    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:35.850261 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846880    4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846882    4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846885    4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846888    4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846890    4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846892    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846908    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846912    4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846915    4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846917    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846920    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846923    4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846925    4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846929    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846932    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846936    4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.846941    4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847446    4494 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847453    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:35.850743 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847458    4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847462    4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847465    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847468    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847471    4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847473    4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847476    4494 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847479    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847481    4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847484    4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847486    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847489    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847492    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847494    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847497    4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847499    4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847502    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847504    4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847507    4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847509    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:35.851255 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847512    4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847514    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847517    4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847519    4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847522    4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847524    4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847527    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847530    4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847532    4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847538    4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847540    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847545    4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847547    4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847550    4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847553    4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847555    4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847558    4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847561    4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847563    4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847566    4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:35.851834 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847568    4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847571    4494 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847573    4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847576    4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847578    4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847581    4494 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847583    4494 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847586    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847588    4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847591    4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847594    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847596    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847599    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847601    4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847603    4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847606    4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847609    4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847611    4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847613    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847619    4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:35.852477 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847622    4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847625    4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847628    4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847632    4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847634    4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847637    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847639    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847642    4494 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847644    4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847647    4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847651    4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847655    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847658    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847660    4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847663    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847665    4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847668    4494 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847670    4494 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847673    4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:35.853038 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847675    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847678    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847680    4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847682    4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847685    4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847688    4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847690    4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847692    4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847695    4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847697    4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847700    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847702    4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847705    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847708    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847712    4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847715    4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847717    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847720    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847722    4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:35.853512 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847725    4494 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847728    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847730    4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847733    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847735    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.847738    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847831    4494 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847838    4494 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847847    4494 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847855    4494 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847861    4494 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847866    4494 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847874    4494 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847879    4494 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847882    4494 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847885    4494 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847889    4494 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847892    4494 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847907    4494 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847911    4494 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847914    4494 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847917    4494 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847921    4494 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:55:35.853993 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847924    4494 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847928    4494 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847931    4494 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847934    4494 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847939    4494 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847943    4494 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847947    4494 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847950    4494 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847953    4494 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847957    4494 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847960    4494 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847963    4494 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847966    4494 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847969    4494 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847972    4494 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847977    4494 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847980    4494 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847983    4494 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847986    4494 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847989    4494 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847992    4494 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.847997    4494 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848000    4494 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848003    4494 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:55:35.854563 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848006    4494 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848009    4494 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848013    4494 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848016    4494 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848019    4494 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848022    4494 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848025    4494 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848029    4494 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848032    4494 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848035    4494 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848037    4494 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848040    4494 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848045    4494 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848052    4494 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848056    4494 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848059    4494 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848062    4494 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848065    4494 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848068    4494 flags.go:64] FLAG: --help="false"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848071    4494 flags.go:64] FLAG: --hostname-override="ip-10-0-130-130.ec2.internal"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848074    4494 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848077    4494 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848080    4494 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848083    4494 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:55:35.855249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848086    4494 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848089    4494 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848092    4494 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848095    4494 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848098    4494 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848101    4494 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848104    4494 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848107    4494 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848110    4494 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848113    4494 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848116    4494 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848119    4494 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848122    4494 flags.go:64] FLAG: --lock-file=""
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848126    4494 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848128    4494 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848131    4494 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848137    4494 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848140    4494 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848144    4494 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848147    4494 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848151    4494 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848155    4494 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848158    4494 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848161    4494 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848166    4494 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:55:35.856033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848169    4494 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848173    4494 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848176    4494 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848179    4494 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848182    4494 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848185    4494 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848188    4494 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848191    4494 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848194    4494 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848202    4494 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848207    4494 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848212    4494 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848217    4494 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848222    4494 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848225    4494 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848228    4494 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848231    4494 flags.go:64] FLAG: --port="10250"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848234    4494 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848237    4494 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f104715efb0af7e7"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848240    4494 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848245    4494 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848248    4494 flags.go:64] FLAG: --register-node="true"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848251    4494 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848255    4494 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848258    4494 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848261    4494 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:55:35.857030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848264    4494 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848269    4494 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848272    4494 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848275    4494 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848278    4494 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848281    4494 flags.go:64] FLAG: --runonce="false"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848284    4494 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848287    4494 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848290    4494 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848293    4494 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848296    4494 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848299    4494 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848302    4494 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848305    4494 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848308    4494 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848311    4494 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848313    4494 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848316    4494 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848319    4494 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848322    4494 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848325    4494 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848330    4494 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848333    4494 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848335    4494 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848339    4494 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:55:35.858096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848342    4494 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848347    4494 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848350    4494 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848353    4494 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848356    4494 flags.go:64] FLAG: --v="2"
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848360    4494 flags.go:64] FLAG: --version="false"
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848364    4494 flags.go:64] FLAG: --vmodule=""
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848367    4494 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848372    4494 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848515    4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848519    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848522    4494 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848525    4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848528    4494 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848531    4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848533    4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848536    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848539    4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848541    4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848544    4494 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848546    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848549    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:35.859226 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848551    4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848554    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848556    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848559    4494 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848561    4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848564    4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848567    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848569    4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848571    4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848574    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848576    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848580    4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848583    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848586    4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848588    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848591    4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848593    4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848596    4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848600    4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:35.860408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848603    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848606    4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848608    4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848611    4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848613    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848616    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848619    4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848621    4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848624    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848626    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848629    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848632    4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848634    4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848637    4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848642    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848645    4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848647    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848650    4494 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848652    4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848655    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:35.861068 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848657    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848659    4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848662    4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848664    4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848668    4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848670    4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848673    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848675    4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848678    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848680    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848683    4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848687    4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848689    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848692    4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848694    4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848697    4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848699    4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848702    4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848705    4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:35.861670 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848708    4494 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848710    4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848712    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848715    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848718    4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848720    4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848723    4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848725    4494 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848727    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848730    4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848733    4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848735    4494 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848737    4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848740    4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848742    4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848744    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848747    4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848750    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848753    4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848755    4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:35.862258 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848758    4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848760    4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848762    4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848766    4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848772    4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848775    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848778    4494 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848782    4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848785    4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848788    4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848790    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848794    4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848796    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848799    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848801    4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:35.862823 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.848828    4494 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848866    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848871    4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848874    4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848878    4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848881    4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848884    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848887    4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848889    4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848892    4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848908    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848911    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848913    4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848916    4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848920    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848922    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848925    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848928    4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848930    4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848933    4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:35.863254 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848935    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848938    4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848941    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848944    4494 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848946    4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848949    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848951    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848954    4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848956    4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848959    4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848962    4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848965    4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848967    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848970    4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848972    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848974    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848977    4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848980    4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848982    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:35.863760 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848984    4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848987    4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848989    4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848992    4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848995    4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848997    4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.848999    4494 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849003    4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849006    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849010    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849013    4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849015    4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849017    4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849020    4494 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849022    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849026    4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849029    4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849031    4494 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849034    4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849036    4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:35.864379 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849039    4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849041    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849044    4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849047    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849049    4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849052    4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849054    4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849057    4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849060    4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849064    4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849068    4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849071    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849074    4494 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849079    4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849085    4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849090    4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849094    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849098    4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849101    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:35.865267 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849107    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849111    4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849114    4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849117    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849119    4494 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849122    4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849124    4494 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849127    4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849131    4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849133    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849136    4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849138    4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849141    4494 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849143    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849145    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849148    4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849151    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849154    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849156    4494 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849159    4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:35.865915 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849161    4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849164    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849166    4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849169    4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849173    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849177    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849181    4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849185    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.849191    4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.853676    4494 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.853697    4494 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.853718    4494 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.853724    4494 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853783    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853789    4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853792    4494 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853795    4494 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853798    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853801    4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:35.866438 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853804    4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853807    4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853810    4494 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853812    4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853815    4494 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853817    4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853820    4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853822    4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853825    4494 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853828    4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853830    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853833    4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853835    4494 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853837    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853841    4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853843    4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853846    4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853848    4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853851    4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853853    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:35.866932 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853856    4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853859    4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853863    4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853868    4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853872    4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853876    4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853879    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853882    4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853884    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853887    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853889    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853892    4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853915    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853919    4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853922    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853925    4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853927    4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853930    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853932    4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:35.867412 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853935    4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853937    4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853940    4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853943    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853945    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853948    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853950    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853953    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853956    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853958    4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853961    4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853963    4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853966    4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853969    4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853971    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853974    4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853976    4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853980    4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853983    4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:35.867927 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853985    4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853988    4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853990    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853992    4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853995    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.853998    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854000    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854003    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854005    4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854008    4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854011    4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854013    4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854015    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854018    4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854020    4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854023    4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854025    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854028    4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854030    4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:35.868395 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854032    4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854035    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854038    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854040    4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854043    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854045    4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854048    4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854050    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854053    4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854055    4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854058    4494 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854060    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854063    4494 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854066    4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854068    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854071    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854074    4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854076    4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854078    4494 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854081    4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:35.868887 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854083    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854085    4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854088    4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.854116    4494 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854267    4494 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854272    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854275    4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854278    4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854280    4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854282    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854285    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854288    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854290    4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854293    4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854296    4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854298    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:35.869575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854301    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854304    4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854307    4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854309    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854312    4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854314    4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854317    4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854319    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854322    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854330    4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854333    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854336    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854340    4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854344    4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854347    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854350    4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854353    4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854356    4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:35.869991 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854359    4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854362    4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854365    4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854367    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854370    4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854372    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854375    4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854378    4494 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854381    4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854384    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854386    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854389    4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854391    4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854394    4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854396    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854399    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854401    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854404    4494 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854406    4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854409    4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:35.870465 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854411    4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854414    4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854417    4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854419    4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854422    4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854425    4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854427    4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854430    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854433    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854435    4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854438    4494 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854440    4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854443    4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854445    4494 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854448    4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854450    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854453    4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854455    4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854457    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854460    4494 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:35.870980 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854463    4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854465    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854468    4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854470    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854473    4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854475    4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854478    4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854480    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854483    4494 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854486    4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854488    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854491    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854493    4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854496    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854498    4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854501    4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854503    4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854506    4494 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854508    4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854511    4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:35.871475 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854514    4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854516    4494 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854518    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854522    4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854526    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854529    4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854532    4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854534    4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854537    4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854539    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854542    4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854544    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854547    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854550    4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854552    4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854555    4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:35.871987 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.854579    4494 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854610    4494 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854614    4494 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854617    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854620    4494 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854623    4494 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854626    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854628    4494 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854631    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854633    4494 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854636    4494 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854638    4494 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854641    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854643    4494 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854646    4494 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854649    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854652    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854654    4494 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854657    4494 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854659    4494 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:35.872377 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854662    4494 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854664    4494 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854666    4494 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854669    4494 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854671    4494 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854674    4494 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854676    4494 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854679    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854683    4494 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854686    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854689    4494 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854691    4494 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854693    4494 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854696    4494 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854698    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854701    4494 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854703    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854706    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854711    4494 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:35.872857 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854713    4494 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854716    4494 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854719    4494 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854721    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854724    4494 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854726    4494 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854729    4494 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854731    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854734    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854737    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854740    4494 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854742    4494 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854745    4494 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854748    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854750    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854753    4494 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854755    4494 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854758    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854760    4494 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:35.873352 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854763    4494 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854765    4494 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854768    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854770    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854773    4494 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854776    4494 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854779    4494 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854783    4494 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854786    4494 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854789    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854792    4494 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854795    4494 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854799    4494 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854802    4494 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854804    4494 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854807    4494 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854810    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854812    4494 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:35.873843 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854815    4494 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854817    4494 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854821    4494 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854824    4494 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854827    4494 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854830    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854832    4494 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854835    4494 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854838    4494 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854840    4494 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854842    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854845    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854847    4494 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854850    4494 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854852    4494 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854855    4494 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854857    4494 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854868    4494 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854871    4494 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854874    4494 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:35.874571 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854877    4494 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854879    4494 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854881    4494 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854884    4494 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854887    4494 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854890    4494 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854910    4494 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854915    4494 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854919    4494 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854923    4494 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:35.854927    4494 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.854991    4494 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.856954    4494 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.857319    4494 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.857675    4494 server.go:1014] "Starting client certificate rotation"
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.857797    4494 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.857846    4494 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:49:56 +0000 UTC" deadline="2028-06-16 07:24:04.422573277 +0000 UTC"
Oct 30 02:55:35.875469 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.857884    4494 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14284h28m28.564692618s"
Oct 30 02:55:35.876223 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.858490    4494 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:55:35.876223 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.860257    4494 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:55:35.876223 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.861785    4494 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:55:35.876223 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.863506    4494 log.go:25] "Validated CRI v1 image API"
Oct 30 02:55:35.876223 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.864320    4494 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:55:35.876223 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.865867    4494 fs.go:135] Filesystem UUIDs: map[44c5f80b-4751-4159-93bc-0d264142889d:/dev/nvme0n1p3 49b7a8ae-bb84-4666-9864-724fa5d7fc51:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 02:55:35.876398 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.865891    4494 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f9167606fbccd6136d34ef05ec6dc82a0dbf080e7834191c193a2f0c39dc9fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f9167606fbccd6136d34ef05ec6dc82a0dbf080e7834191c193a2f0c39dc9fd/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15f683ad8289c1a7eab25fbfddc39ff6ec5f73becd79c0e5e6bb7dc6ccc71690/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15f683ad8289c1a7eab25fbfddc39ff6ec5f73becd79c0e5e6bb7dc6ccc71690/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16fe60e79e0c3502099ee29d15fd36b92eb474340d9795b90ce55e94945812fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16fe60e79e0c3502099ee29d15fd36b92eb474340d9795b90ce55e94945812fc/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25da7a4ec4f13430449973faad5114f078e866c970a54b5c43bfb402da790236/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25da7a4ec4f13430449973faad5114f078e866c970a54b5c43bfb402da790236/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3715466adbb270ef72769304c6f8bbad7bb59bd00ae8126b6cc0858d41a46132/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3715466adbb270ef72769304c6f8bbad7bb59bd00ae8126b6cc0858d41a46132/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49fabea07507cca26bdcf3a6577a1a06fa8fd93cf4235af5c4e726e14e5b0f13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49fabea07507cca26bdcf3a6577a1a06fa8fd93cf4235af5c4e726e14e5b0f13/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/504a98366245e55532fe800e409e5e32641a6195e9f8edcef0453c44eefd4c15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/504a98366245e55532fe800e409e5e32641a6195e9f8edcef0453c44eefd4c15/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e14ae5ecfb06b378739119d38b03970e4b299c88e268634c12b0d6974ead0f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e14ae5ecfb06b378739119d38b03970e4b299c88e268634c12b0d6974ead0f8/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e173113865d2ad09afa50c01803a24dd78262de029f3d578c38abdf0c300a3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e173113865d2ad09afa50c01803a24dd78262de029f3d578c38abdf0c300a3a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d57fb178fd29aecbb79ee7c69db7d63e5010270fe678c2a5607032b0618c730/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d57fb178fd29aecbb79ee7c69db7d63e5010270fe678c2a5607032b0618c730/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8395834fd92499117320b142cc41fccfb4e8caf90bf4c3328194e68a1cef2a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8395834fd92499117320b142cc41fccfb4e8caf90bf4c3328194e68a1cef2a5/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b47016b6f5df54ae2183eb9ae086099c839f58a485f50abed70866ca5bb50fd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b47016b6f5df54ae2183eb9ae086099c839f58a485f50abed70866ca5bb50fd4/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b50c6dc549d18e7f6dbca52525d6276f51119940d279437939a7abb125b5d057/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b50c6dc549d18e7f6dbca52525d6276f51119940d279437939a7abb125b5d057/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8fd7b271a8f40ef7dbe3a14e0b09cd73a6ad62f6722a3b7f2ad86e13aa5ec10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8fd7b271a8f40ef7dbe3a14e0b09cd73a6ad62f6722a3b7f2ad86e13aa5ec10/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e956f9e234a728c0914d1dea985a699d79c229efa2ecdb6f4458e3aadf6a4fcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e956f9e234a728c0914d1dea985a699d79c229efa2ecdb6f4458e3aadf6a4fcd/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d303f31-40f2-46fc-9bb1-73ba2851a5ec/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0d303f31-40f2-46fc-9bb1-73ba2851a5ec/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1db5896f-bc1f-4965-9282-a344f5ad8e03/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1db5896f-bc1f-4965-9282-a344f5ad8e03/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1db5896f-bc1f-4965-9282-a344f5ad8e03/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1db5896f-bc1f-4965-9282-a344f5ad8e03/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1db5896f-bc1f-4965-9282-a344f5ad8e03/volumes/kubernetes.io~projected/kube-api-access-b5txc:{mountpoint:/var/lib/kubelet/pods/1db5896f-bc1f-4965-9282-a344f5ad8e03/volumes/kubernetes.io~projected/kube-api-access-b5txc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23330a95-b468-4fa4-84d7-d5d58fbdf1cb/volumes/kubernetes.io~projected/kube-api-access-lkz58:{mountpoint:/var/lib/kubelet/pods/23330a95-b468-4fa4-84d7-d5d58fbdf1cb/volumes/kubernetes.io~projected/kube-api-access-lkz58 major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27811319-29a6-459b-a1e3-8325c6cc2274/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/27811319-29a6-459b-a1e3-8325c6cc2274/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27811319-29a6-459b-a1e3-8325c6cc2274/volumes/kubernetes.io~projected/kube-api-access-2b86p:{mountpoint:/var/lib/kubelet/pods/27811319-29a6-459b-a1e3-8325c6cc2274/volumes/kubernetes.io~projected/kube-api-access-2b86p major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27811319-29a6-459b-a1e3-8325c6cc2274/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/27811319-29a6-459b-a1e3-8325c6cc2274/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a73c32c-6071-4eb7-8215-171cb490ee56/volumes/kubernetes.io~projected/kube-api-access-xr48h:{mountpoint:/var/lib/kubelet/pods/2a73c32c-6071-4eb7-8215-171cb490ee56/volumes/kubernetes.io~projected/kube-api-access-xr48h major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c7ca2bd-121d-4ef9-ae00-4a139497c91d/volumes/kubernetes.io~projected/kube-api-access-c69r7:{mountpoint:/var/lib/kubelet/pods/2c7ca2bd-121d-4ef9-ae00-4a139497c91d/volumes/kubernetes.io~projected/kube-api-access-c69r7 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c7ca2bd-121d-4ef9-ae00-4a139497c91d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2c7ca2bd-121d-4ef9-ae00-4a139497c91d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65/volumes/kubernetes.io~projected/kube-api-access-j9cqv:{mountpoint:/var/lib/kubelet/pods/32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65/volumes/kubernetes.io~projected/kube-api-access-j9cqv major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33044703-7fec-4a2f-a1c7-cde0e8dd50bc/volumes/kubernetes.io~projected/kube-api-access-9m4tt:{mountpoint:/var/lib/kubelet/pods/33044703-7fec-4a2f-a1c7-cde0e8dd50bc/volumes/kubernetes.io~projected/kube-api-access-9m4tt major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/603f9058-eea9-4899-872c-503c7111dcc7/volumes/kubernetes.io~projected/kube-api-access-tjv7m:{mountpoint:/var/lib/kubelet/pods/603f9058-eea9-4899-872c-503c7111dcc7/volumes/kubernetes.io~projected/kube-api-access-tjv7m major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1/volumes/kubernetes.io~projected/kube-api-access-4nflm:{mountpoint:/var/lib/kubelet/pods/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1/volumes/kubernetes.io~projected/kube-api-access-4nflm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5/volumes/kubernetes.io~projected/kube-api-access-d2hwt:{mountpoint:/var/lib/kubelet/pods/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5/volumes/kubernetes.io~projected/kube-api-access-d2hwt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1df94c2-8be5-42df-9d04-7569e45f7648/volumes/kubernetes.io~projected/kube-api-access-kbnn8:{mountpoint:/var/lib/kubelet/pods/a1df94c2-8be5-42df-9d04-7569e45f7648/volumes/kubernetes.io~projected/kube-api-access-kbnn8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac850265-e643-4365-b318-3507b8c838b9/volumes/kubernetes.io~projected/kube-api-access-tfjxv:{mountpoint:/var/lib/kubelet/pods/ac850265-e643-4365-b318-3507b8c838b9/volumes/kubernetes.io~projected/kube-api-access-tfjxv major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf0213f1-3092-4a53-ba3a-1cec86701275/volumes/kubernetes.io~projected/kube-api-access-ttxgw:{mountpoint:/var/lib/kubelet/pods/bf0213f1-3092-4a53-ba3a-1cec86701275/volumes/kubernetes.io~projected/kube-api-access-ttxgw major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8c74de0-5c52-4e2e-9f03-3b982a4e7667/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c8c74de0-5c52-4e2e-9f03-3b982a4e7667/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e18c7963-1218-4b83-837b-391b7d3edd22/volumes/kubernetes.io~projected/kube-api-access-p8vxl:{mountpoint:/var/lib/kubelet/pods/e18c7963-1218-4b83-837b-391b7d3edd22/volumes/kubernetes.io~projected/kube-api-access-p8vxl major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2/volumes/kubernetes.io~projected/kube-api-access-v45bk:{mountpoint:/var/lib/kubelet/pods/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2/volumes/kubernetes.io~projected/kube-api-access-v45bk major:0 minor:204 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/eeb02179d127e87886469097248bc0a9d268a6093e91ae86465b732a49f927fc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/fdf1b46c2694c89f7b9699358bbf6e8860c50704f012d6dc489a9f2125355caa/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/52cbc7158e510fa8d8f392afd39de3f67a4e0df2bfde65ad240f93dfdf787af4/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c398ce860496cd84a3b8518a412b5cf942e8fac8e94ef9771983ffd96555d8d9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/71a1f7a272f0785a5a748af02b70fd11db007506e7ace714131a0c489609785d/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4c315724b648a5aad75856b0a25b607e151f68b9fa57327fbc480b6c583d47a0/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ddff95a3da3ca39bb8b37178ff08730c445e29491377932010065d04ca6202e4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ecac97466631c9b389ec61936ea83d0c5692dc4206589785cd87550a0cece620/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/43483cc1aaf07bb8ef3406cfd7e47aeff4a567f2efc60dbd46a173773a25a653/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/71647fb558227000c668e5a3deaa0822a1f5e6ec7a29c788326ea14d3c038427/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/beda789150d8286085f7ac089204c659beb03f22549c28d64ad3d18c3a9b19ae/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/51d717e73b9b744758491e416e7e145417b33a99381094cdbd3861fb130f4756/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/54f972d9fb7bd628ee7527c96c072f5bbd9fee63bbe221cdffc015547b596781/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/f59b8ddb283df38848e4a2e2f6dbe1b8ca891a1f0b88c1bb384a0d2eb381a763/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/0eaa67784b258ae62e11ab6c51c3832955829b9801b10387d507730e13d8ccba/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8e7a448a7542fc3f36f932656f3518d327d4582398d0afdecdd49ff168839e3c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/10bcc9c60297aef1a664a3735a94386e29c2c48f058dd8f63924362c707d023e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a8165d42c290c87f15bb32c6d277635daf276135f1f53822a107bd694f6dccac/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/459fa1c41b755930d7fb993ce69f488f055a11659faee6ea0a96f7ed79833890/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9dcdad7f3027aacd3c64d00b3f4532250744da1c5d9d5e2fd5af181cc588c582/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/adfb7612e8b2655a6d8839ac9f0b817ca61b9653ccbcea6d6bd388e45dbb85ae/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/dc61cfc504f4490ccf0179c371aae7559adcb162be632bbd7dc3398ed2f4639c/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/58601018eaab3475a9767ee68aa3ac03cb97095a49e68cc859f749d767ec4429/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/9bccc0f4d2233dd6d3e628a0ec845902a938e75e98aad11e94367064d94fc089/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/f26f518bc9144a3511a0be3b68cbbb729a668b4eaebbf8ba0226b6ba9fc20453/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/f585d346fd364972162c34b2a806395465412e11dcce12e9f92e79b05559f727/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/df5f96c5e4a1f291dec39cd9455638332471e1b1a6b01e398a6a0396a4fc13c2/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/c4e46340910574ad3d735081ae454c125cafae19a2a67dccb61d154c5c6011cb/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/224f9f756a722999fbcea3375221a9e19928ae639657d4e1478e914d89f880d8/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/7d175508d368658d6d9785f35c6f650602411a822950a8e5deeb306f9ed3f203/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/acea9d2254f2a6a218951937194a5d84e5cbc8731bbac216917668992ae388c7/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/4052811cbce30e88aa022d7829033f14790c797896949308a790dd07ca292e66/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d22e1f4792f16218bf81446889812d5891d49af3b50a83f175f36f14844e1dac/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/18de07ff65e4e88c63aba1f73d709934cfa969ab53f901f0e23e4c4e49b3e32a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/4f709ddc1cb8c72386adda76b6e750c7665a910a51cfca1deaced7bfca1842b8/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/dcf1e66958ea1d33a46508a5d26f11ad573320b8335864525cf58441f4cf36fd/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/24900321cdd80c8054982cffe03320b1618dc5d5a37062b104b9a5386aa87b57/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/0f704f702b93a2672f9cc09ec3d2c7d896e58c66c9360052853dc77545a3ce55/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/1a6ef30fd2e195aefedc10d3ef8693ee8c3040a2f9b7f30e19031ca866e7df36/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/17f10f9353b61d6373a35c4666d5b6d32d6e4e25bab027772ef43e4a900cad21/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/86731aa90757ac0a44a62bfe1a10a9df7676a6288c82ac7471d441f464b7e4f6/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.872113    4494 manager.go:217] Machine: {Timestamp:2026-10-30 02:55:35.871632582 +0000 UTC m=+0.077812810 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3080039 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ae4d5b6a527d0b677103a0ad28fa6 SystemUUID:ec2ae4d5-b6a5-27d0-b677-103a0ad28fa6 BootID:53a0ee35-d5e0-4490-a2b9-ecd29de1b4ed Filesystems:[{Device:/var/lib/kubelet/pods/1db5896f-bc1f-4965-9282-a344f5ad8e03/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c8c74de0-5c52-4e2e-9f03-3b982a4e7667/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8fd7b271a8f40ef7dbe3a14e0b09cd73a6ad62f6722a3b7f2ad86e13aa5ec10/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e173113865d2ad09afa50c01803a24dd78262de029f3d578c38abdf0c300a3a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27811319-29a6-459b-a1e3-8325c6cc2274/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/27811319-29a6-459b-a1e3-8325c6cc2274/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/603f9058-eea9-4899-872c-503c7111dcc7/volumes/kubernetes.io~projected/kube-api-access-tjv7m DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f9167606fbccd6136d34ef05ec6dc82a0dbf080e7834191c193a2f0c39dc9fd/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49fabea07507cca26bdcf3a6577a1a06fa8fd93cf4235af5c4e726e14e5b0f13/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1db5896f-bc1f-4965-9282-a344f5ad8e03/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/33044703-7fec-4a2f-a1c7-cde0e8dd50bc/volumes/kubernetes.io~projected/kube-api-access-9m4tt DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e956f9e234a728c0914d1dea985a699d79c229efa2ecdb6f4458e3aadf6a4fcd/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1/volumes/kubernetes.io~projected/kube-api-access-4nflm DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5/volumes/kubernetes.io~projected/kube-api-access-d2hwt DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65/volumes/kubernetes.io~projected/kube-api-access-j9cqv DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1db5896f-bc1f-4965-9282-a344f5ad8e03/volumes/kubernetes.io~projected/kube-api-access-b5txc DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2a73c32c-6071-4eb7-8215-171cb490ee56/volumes/kubernetes.io~projected/kube-api-access-xr48h DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c7ca2bd-121d-4ef9-ae00-4a139497c91d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8395834fd92499117320b142cc41fccfb4e8caf90bf4c3328194e68a1cef2a5/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b47016b6f5df54ae2183eb9ae086099c839f58a485f50abed70866ca5bb50fd4/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3715466adbb270ef72769304c6f8bbad7bb59bd00ae8126b6cc0858d41a46132/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b50c6dc549d18e7f6dbca52525d6276f51119940d279437939a7abb125b5d057/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d303f31-40f2-46fc-9bb1-73ba2851a5ec/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:231 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ac850265-e643-4365-b318-3507b8c838b9/volumes/kubernetes.io~projected/kube-api-access-tfjxv DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e14ae5ecfb06b378739119d38b03970e4b299c88e268634c12b0d6974ead0f8/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/504a98366245e55532fe800e409e5e32641a6195e9f8edcef0453c44eefd4c15/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16fe60e79e0c3502099ee29d15fd36b92eb474340d9795b90ce55e94945812fc/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27811319-29a6-459b-a1e3-8325c6cc2274/volumes/kubernetes.io~projected/kube-api-access-2b86p DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e18c7963-1218-4b83-837b-391b7d3edd22/volumes/kubernetes.io~projected/kube-api-access-p8vxl DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/23330a95-b468-4fa4-84d7-d5d58fbdf1cb/volumes/kubernetes.io~projected/kube-api-access-lkz58 DeviceMajor:0 DeviceMinor:236 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a1df94c2-8be5-42df-9d04-7569e45f7648/volumes/kubernetes.io~projected/kube-api-access-kbnn8 DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25da7a4ec4f13430449973faad5114f078e866c970a54b5c43bfb402da790236/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d57fb178fd29aecbb79ee7c69db7d63e5010270fe678c2a5607032b0618c730/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf0213f1-3092-4a53-ba3a-1cec86701275/volumes/kubernetes.io~projected/kube-api-access-ttxgw DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2c7ca2bd-121d-4ef9-ae00-4a139497c91d/volumes/kubernetes.io~projected/kube-api-access-c69r7 DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2/volumes/kubernetes.io~projected/kube-api-access-v45bk DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15f683ad8289c1a7eab25fbfddc39ff6ec5f73becd79c0e5e6bb7dc6ccc71690/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:15f683ad8289c1a MacAddress:76:7e:be:ec:5d:7d Speed:10000 Mtu:8901} {Name:49fabea07507cca MacAddress:d2:3c:c5:c0:08:69 Speed:10000 Mtu:8901} {Name:504a98366245e55 MacAddress:da:d2:f4:99:41:58 Speed:10000 Mtu:8901} {Name:a8395834fd92499 MacAddress:1e:19:ec:34:93:0f Speed:10000 Mtu:8901} {Name:b50c6dc549d18e7 MacAddress:ce:c1:9b:42:77:89 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:5e:09:d3 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:bf:ed:7f:1e:00 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:5e:09:d3 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:62:21:ba:4c:01:87 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:cc:44:74:6e:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.872378    4494 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.872425    4494 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.872757    4494 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.872785    4494 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.872937    4494 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.872945    4494 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.872957    4494 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.872973    4494 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.873078    4494 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.873272    4494 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.873388    4494 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.873424    4494 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.873434    4494 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.873446    4494 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.873455    4494 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.873470    4494 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.875632    4494 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.875652    4494 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:55:35.876576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.876460    4494 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:55:35.878402 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878388    4494 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:55:35.878439 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878403    4494 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:55:35.878515 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878505    4494 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 02:55:35.878705 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878694    4494 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:55:35.878739 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878724    4494 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:55:35.878823 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878814    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:55:35.878852 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878829    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:55:35.878852 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878836    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:55:35.878852 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878842    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:55:35.878852 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878848    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:55:35.878852 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878853    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:55:35.879035 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878859    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:55:35.879035 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878879    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:55:35.879035 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878885    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:55:35.879035 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878891    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:55:35.879035 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878917    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:55:35.879035 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878928    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:55:35.879035 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878951    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:55:35.879035 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.878957    4494 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:55:35.882364 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.882348    4494 server.go:1263] "Started kubelet"
Oct 30 02:55:35.882556 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.882508    4494 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:55:35.882609 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.882581    4494 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:55:35.882917 ip-10-0-130-130 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:55:35.884040 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.882478    4494 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:55:35.884040 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.882965    4494 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:55:35.884183 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.884165    4494 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.885540    4494 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.885577    4494 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.885669    4494 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.885679    4494 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:35.885686    4494 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-130.ec2.internal\" not found"
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.885823    4494 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.885625    4494 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:58 +0000 UTC" deadline="2028-05-17 12:29:11.445263559 +0000 UTC"
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.886078    4494 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13569h33m35.559189879s"
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.887916    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cb55cf-0333-4d56-a6e0-4c13d0bd70a5" volumeName="kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-daemon-config" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.887978    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cb55cf-0333-4d56-a6e0-4c13d0bd70a5" volumeName="kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888002    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e18c7963-1218-4b83-837b-391b7d3edd22" volumeName="kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-service-ca-bundle" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888015    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63a54a5-8e93-4f23-a7da-a3a22d63f2b2" volumeName="kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888027    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d303f31-40f2-46fc-9bb1-73ba2851a5ec" volumeName="kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888045    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27811319-29a6-459b-a1e3-8325c6cc2274" volumeName="kubernetes.io/secret/27811319-29a6-459b-a1e3-8325c6cc2274-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888059    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a73c32c-6071-4eb7-8215-171cb490ee56" volumeName="kubernetes.io/projected/2a73c32c-6071-4eb7-8215-171cb490ee56-kube-api-access-xr48h" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888079    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33044703-7fec-4a2f-a1c7-cde0e8dd50bc" volumeName="kubernetes.io/projected/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-kube-api-access-9m4tt" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888093    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ca98f80-def1-4e19-8bc6-0f8327e2a5f1" volumeName="kubernetes.io/projected/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-kube-api-access-4nflm" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888113    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e18c7963-1218-4b83-837b-391b7d3edd22" volumeName="kubernetes.io/empty-dir/e18c7963-1218-4b83-837b-391b7d3edd22-archives-path" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888126    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db5896f-bc1f-4965-9282-a344f5ad8e03" volumeName="kubernetes.io/projected/1db5896f-bc1f-4965-9282-a344f5ad8e03-kube-api-access-b5txc" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888154    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27811319-29a6-459b-a1e3-8325c6cc2274" volumeName="kubernetes.io/projected/27811319-29a6-459b-a1e3-8325c6cc2274-kube-api-access-2b86p" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888181    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="603f9058-eea9-4899-872c-503c7111dcc7" volumeName="kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888198    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1df94c2-8be5-42df-9d04-7569e45f7648" volumeName="kubernetes.io/projected/a1df94c2-8be5-42df-9d04-7569e45f7648-kube-api-access-kbnn8" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888223    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8c74de0-5c52-4e2e-9f03-3b982a4e7667" volumeName="kubernetes.io/secret/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-agent-certs" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888238    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e18c7963-1218-4b83-837b-391b7d3edd22" volumeName="kubernetes.io/projected/e18c7963-1218-4b83-837b-391b7d3edd22-kube-api-access-p8vxl" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888250    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63a54a5-8e93-4f23-a7da-a3a22d63f2b2" volumeName="kubernetes.io/projected/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-api-access-v45bk" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888274    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c7ca2bd-121d-4ef9-ae00-4a139497c91d" volumeName="kubernetes.io/secret/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-cert" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888285    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="603f9058-eea9-4899-872c-503c7111dcc7" volumeName="kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888302    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="603f9058-eea9-4899-872c-503c7111dcc7" volumeName="kubernetes.io/projected/603f9058-eea9-4899-872c-503c7111dcc7-kube-api-access-tjv7m" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888322    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8c74de0-5c52-4e2e-9f03-3b982a4e7667" volumeName="kubernetes.io/configmap/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-konnectivity-ca" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888334    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db5896f-bc1f-4965-9282-a344f5ad8e03" volumeName="kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-tuned" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888345    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27811319-29a6-459b-a1e3-8325c6cc2274" volumeName="kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888361    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac850265-e643-4365-b318-3507b8c838b9" volumeName="kubernetes.io/empty-dir/ac850265-e643-4365-b318-3507b8c838b9-tmp-dir" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888372    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ca98f80-def1-4e19-8bc6-0f8327e2a5f1" volumeName="kubernetes.io/configmap/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-iptables-alerter-script" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888399    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf0213f1-3092-4a53-ba3a-1cec86701275" volumeName="kubernetes.io/configmap/bf0213f1-3092-4a53-ba3a-1cec86701275-config-volume" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888412    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e18c7963-1218-4b83-837b-391b7d3edd22" volumeName="kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888423    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63a54a5-8e93-4f23-a7da-a3a22d63f2b2" volumeName="kubernetes.io/empty-dir/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-data-volume" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888443    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1db5896f-bc1f-4965-9282-a344f5ad8e03" volumeName="kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-tmp" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888460    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23330a95-b468-4fa4-84d7-d5d58fbdf1cb" volumeName="kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888473    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27811319-29a6-459b-a1e3-8325c6cc2274" volumeName="kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-env-overrides" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888489    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65" volumeName="kubernetes.io/projected/32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65-kube-api-access-j9cqv" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888501    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac850265-e643-4365-b318-3507b8c838b9" volumeName="kubernetes.io/projected/ac850265-e643-4365-b318-3507b8c838b9-kube-api-access-tfjxv" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888518    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="603f9058-eea9-4899-872c-503c7111dcc7" volumeName="kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888534    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cb55cf-0333-4d56-a6e0-4c13d0bd70a5" volumeName="kubernetes.io/projected/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-kube-api-access-d2hwt" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888550    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf0213f1-3092-4a53-ba3a-1cec86701275" volumeName="kubernetes.io/empty-dir/bf0213f1-3092-4a53-ba3a-1cec86701275-tmp-dir" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888564    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf0213f1-3092-4a53-ba3a-1cec86701275" volumeName="kubernetes.io/projected/bf0213f1-3092-4a53-ba3a-1cec86701275-kube-api-access-ttxgw" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888580    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27811319-29a6-459b-a1e3-8325c6cc2274" volumeName="kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-config" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888600    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c7ca2bd-121d-4ef9-ae00-4a139497c91d" volumeName="kubernetes.io/projected/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-kube-api-access-c69r7" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888615    4494 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33044703-7fec-4a2f-a1c7-cde0e8dd50bc" volumeName="kubernetes.io/configmap/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-serviceca" seLinuxMountContext=""
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888627    4494 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:55:35.888795 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.888635    4494 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:55:35.892334 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.892303    4494 factory.go:55] Registering systemd factory
Oct 30 02:55:35.892495 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.892483    4494 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:55:35.892951 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.892933    4494 factory.go:153] Registering CRI-O factory
Oct 30 02:55:35.892951 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.892951    4494 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:55:35.893096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.893053    4494 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:55:35.893096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.893080    4494 factory.go:103] Registering Raw factory
Oct 30 02:55:35.893096 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.893095    4494 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:55:35.893540 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.893528    4494 manager.go:319] Starting recovery of all containers
Oct 30 02:55:35.919124 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.919067    4494 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:55:35.920471 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.920445    4494 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:55:35.920471 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.920473    4494 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:55:35.920642 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.920499    4494 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:55:35.920642 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:35.920541    4494 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 02:55:35.927524 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.927486    4494 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="07bb3e394e9db5e19921cd49bc377d57a9bce18a77f3120721ed9b03deab9128" exitCode=0
Oct 30 02:55:35.927524 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.927523    4494 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="aa20bb1117040ce4ff2876bde33cadbe24a7a2b50c0eca7c82aea19237e498cb" exitCode=0
Oct 30 02:55:35.927686 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.927537    4494 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="bcec0d38b085829af64dd83e30eff9a049b7e373400035b4d0f19059a88e92c5" exitCode=0
Oct 30 02:55:35.927686 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.927550    4494 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="5b9cefec968f60260836eb0576e0f29e4e28ba75b44d8430e4c6a7373d5d8845" exitCode=0
Oct 30 02:55:35.927686 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.927563    4494 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="16753f30a8904d7eaad1485a0615f62e2d76b4f10970a116cebb3f4c9830d221" exitCode=0
Oct 30 02:55:35.927686 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.927575    4494 generic.go:356] "Generic (PLEG): container finished" podID="603f9058-eea9-4899-872c-503c7111dcc7" containerID="f005b28ce8da21e606b0f3c721b4d4bc3a8ecdfd4efe0810e591715676eacbb8" exitCode=0
Oct 30 02:55:35.940565 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.940538    4494 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 02:55:35.941162 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.941132    4494 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:35.941508 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.941481    4494 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:35.953996 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.953968    4494 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:35.954567 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.954541    4494 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:35.957345 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.957310    4494 generic.go:356] "Generic (PLEG): container finished" podID="2464e4e1bac4126b7f6dfa935cfd24ca" containerID="6264fc73ab0dde912ae68edc2f30144e219c05ba82303cc30d25c11354ff5c95" exitCode=0
Oct 30 02:55:35.966744 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.966720    4494 manager.go:324] Recovery completed
Oct 30 02:55:35.981958 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.981932    4494 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:55:35.981958 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.981952    4494 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:55:35.982132 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.981972    4494 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:55:35.982132 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.982112    4494 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 02:55:35.982132 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.982122    4494 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 02:55:35.982234 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.982138    4494 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 02:55:35.982234 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.982143    4494 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 02:55:35.982234 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.982148    4494 policy_none.go:50] "Start"
Oct 30 02:55:35.982234 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.982157    4494 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:55:35.982234 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.982165    4494 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:55:35.982359 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.982295    4494 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 02:55:35.982359 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.982302    4494 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 02:55:35.982359 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.982308    4494 policy_none.go:44] "Start"
Oct 30 02:55:35.986344 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.986262    4494 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:55:35.989030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.989003    4494 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:55:35.989949 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:35.989037    4494 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:55:35.990014 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.989953    4494 server.go:87] "Starting device plugin registration server"
Oct 30 02:55:35.990216 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.990206    4494 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:55:35.990249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.990218    4494 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:55:35.990330 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.990307    4494 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:55:35.990438 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.990398    4494 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:55:35.990438 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.990425    4494 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:55:35.990438 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.990429    4494 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:55:35.990573 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.990483    4494 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:55:35.990417333Z","UUID":"504f864c-a287-4c44-9f20-6ba2342ea380","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:55:35.991963 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:35.991937    4494 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:55:35.994540 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.994514    4494 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:55:35.994695 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:35.994557    4494 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:55:36.020992 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.020943    4494 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"]
Oct 30 02:55:36.021496 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.021444    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal" event={"ID":"2464e4e1bac4126b7f6dfa935cfd24ca","Type":"ContainerStarted","Data":"bc4115256dcc06419bf50ce7826ff76abeb44ca935181e9b15f7df316e1b0e71"}
Oct 30 02:55:36.021607 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.021497    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal" event={"ID":"2464e4e1bac4126b7f6dfa935cfd24ca","Type":"ContainerDied","Data":"6264fc73ab0dde912ae68edc2f30144e219c05ba82303cc30d25c11354ff5c95"}
Oct 30 02:55:36.021607 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.021515    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal" event={"ID":"2464e4e1bac4126b7f6dfa935cfd24ca","Type":"ContainerStarted","Data":"b47016b6f5df54ae2183eb9ae086099c839f58a485f50abed70866ca5bb50fd4"}
Oct 30 02:55:36.021607 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.021530    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal" event={"ID":"868aae1459799721b266e5e297e80105","Type":"ContainerStarted","Data":"235f54fc267f63fec7acd59a6bcc5bf4007bb3fe32ed2ee6eda0803643a7006e"}
Oct 30 02:55:36.021607 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.021549    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal" event={"ID":"868aae1459799721b266e5e297e80105","Type":"ContainerStarted","Data":"16fe60e79e0c3502099ee29d15fd36b92eb474340d9795b90ce55e94945812fc"}
Oct 30 02:55:36.021607 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.021546    4494 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.021773 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.021626    4494 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.049214 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.049143    4494 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:36.049359 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:36.049234    4494 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-130.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.049359 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.049256    4494 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:36.049359 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:36.049302    4494 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.090776 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.090738    4494 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:55:36.092331 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.092312    4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:55:36.092458 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.092349    4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:55:36.092458 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.092363    4494 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-130.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:55:36.092458 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.092406    4494 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.127521 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.127488    4494 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.127652 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.127572    4494 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.141787 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.141755    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.141932 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.141788    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.141932 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.141809    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/868aae1459799721b266e5e297e80105-config\") pod \"kube-apiserver-proxy-ip-10-0-130-130.ec2.internal\" (UID: \"868aae1459799721b266e5e297e80105\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.242571 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.242481    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.242571 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.242520    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.242571 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.242548    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/868aae1459799721b266e5e297e80105-config\") pod \"kube-apiserver-proxy-ip-10-0-130-130.ec2.internal\" (UID: \"868aae1459799721b266e5e297e80105\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.242790 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.242594    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/868aae1459799721b266e5e297e80105-config\") pod \"kube-apiserver-proxy-ip-10-0-130-130.ec2.internal\" (UID: \"868aae1459799721b266e5e297e80105\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.242790 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.242595    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.242790 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.242597    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2464e4e1bac4126b7f6dfa935cfd24ca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" (UID: \"2464e4e1bac4126b7f6dfa935cfd24ca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.874126 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.874088    4494 apiserver.go:52] "Watching apiserver"
Oct 30 02:55:36.932480 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.932430    4494 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:55:36.933949 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.933880    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-xltm9","openshift-dns/node-resolver-vsx5v","openshift-dns/dns-default-zxwqk","openshift-ingress-canary/ingress-canary-2d7p7","openshift-insights/insights-runtime-extractor-pgwbh","openshift-network-diagnostics/network-check-target-rjwmx","kube-system/konnectivity-agent-m2tdr","openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7","openshift-multus/network-metrics-daemon-ndbtf","openshift-multus/multus-g5vjw","openshift-image-registry/node-ca-f8mgk","openshift-insights/periodic-gathering-clqt4-925xq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal","openshift-cluster-node-tuning-operator/tuned-p6dt8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t","openshift-network-operator/iptables-alerter-6j76z","kube-system/global-pull-secret-syncer-ddlpl","openshift-multus/multus-additional-cni-plugins-cspkf"]
Oct 30 02:55:36.938390 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.938361    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:36.939930 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.939887    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:36.940780 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.940758    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:36.954234 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954203    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:55:36.954559 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954539    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bckpk\""
Oct 30 02:55:36.954650 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954568    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gpqv\""
Oct 30 02:55:36.954716 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954666    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tpj88\""
Oct 30 02:55:36.954716 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954676    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8ljf8\""
Oct 30 02:55:36.954716 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954705    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9qpgh\""
Oct 30 02:55:36.954846 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954713    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kn7z4\""
Oct 30 02:55:36.954846 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954743    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mk2hv\""
Oct 30 02:55:36.954846 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954761    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:55:36.954846 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954748    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:55:36.954846 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954780    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kdtjd\""
Oct 30 02:55:36.955069 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954855    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.955069 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.954927    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.955069 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955007    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.955069 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955063    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.955211 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955071    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:55:36.955211 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955098    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.955211 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955135    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.955211 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955193    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.955337 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955223    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.955337 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955233    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:55:36.955337 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955243    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.955337 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955276    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:55:36.955337 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955281    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:55:36.955337 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955286    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.955337 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955270    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.955337 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955316    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.955337 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.955339    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:55:36.959324 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959303    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:55:36.959423 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959341    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.959423 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959358    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:55:36.959423 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959359    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.959423 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959388    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:55:36.959423 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959397    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.959624 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959433    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:55:36.959624 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959521    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.959624 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959535    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:55:36.959719 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959625    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-chfb9\""
Oct 30 02:55:36.959749 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959722    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:55:36.959784 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959767    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.959958 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959942    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vzrlw\""
Oct 30 02:55:36.960030 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.959968    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fdxtg\""
Oct 30 02:55:36.960131 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.960118    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d8b4w\""
Oct 30 02:55:36.964273 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.964247    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:55:36.966354 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.966332    4494 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.966464 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.966408    4494 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.967037 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.967016    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:55:36.967357 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.967288    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-chr82\""
Oct 30 02:55:36.967530 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.967424    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ft4zm\""
Oct 30 02:55:36.967530 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.967464    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:55:36.967530 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.967482    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.967530 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.967494    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f2mz6\""
Oct 30 02:55:36.967963 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.967941    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 02:55:36.968037 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.968026    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.968184 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.968165    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.968289 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.968165    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 02:55:36.968437 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.968328    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:55:36.970998 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.969669    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 02:55:36.971645 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.971625    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:55:36.971761 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.971655    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:55:36.971761 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.971675    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:55:36.971761 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.971654    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xzg75\""
Oct 30 02:55:36.971761 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.971696    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9wgn\""
Oct 30 02:55:36.987353 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.987319    4494 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:55:36.989210 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.989193    4494 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:36.989273 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:36.989243    4494 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-130.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:36.989818 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:36.989805    4494 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:36.989869 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:36.989844    4494 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-130.ec2.internal"
Oct 30 02:55:37.045250 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045214    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-host\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.045250 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045251    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-dbus\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:37.045459 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045274    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-service-ca-bundle\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:37.045459 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045292    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c69r7\" (UniqueName: \"kubernetes.io/projected/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-kube-api-access-c69r7\") pod \"ingress-canary-2d7p7\" (UID: \"2c7ca2bd-121d-4ef9-ae00-4a139497c91d\") " pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:37.045459 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045355    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0213f1-3092-4a53-ba3a-1cec86701275-tmp-dir\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:37.045459 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045401    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.045616 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045477    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf0213f1-3092-4a53-ba3a-1cec86701275-tmp-dir\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:37.045616 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045508    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-netns\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.045616 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045535    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-var-lib-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.045616 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045577    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b86p\" (UniqueName: \"kubernetes.io/projected/27811319-29a6-459b-a1e3-8325c6cc2274-kube-api-access-2b86p\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.045616 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045601    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxgw\" (UniqueName: \"kubernetes.io/projected/bf0213f1-3092-4a53-ba3a-1cec86701275-kube-api-access-ttxgw\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:37.045868 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045660    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-host-slash\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:55:37.045868 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045705    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.045868 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045742    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-var-lib-kubelet\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.045868 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045765    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-log-socket\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.045868 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045791    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.045868 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045837    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-netd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.046185 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045924    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-system-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.046185 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.045970    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.046185 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046005    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-trusted-ca-bundle\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:37.046185 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046038    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45bk\" (UniqueName: \"kubernetes.io/projected/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-api-access-v45bk\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.046185 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046056    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-service-ca-bundle\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:37.046185 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046065    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-kubelet\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.046185 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046092    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-systemd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.046185 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046134    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-ovn\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.046482 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046212    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27811319-29a6-459b-a1e3-8325c6cc2274-ovn-node-metrics-cert\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.046482 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046242    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-script-lib\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.046482 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046288    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.046482 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046316    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-registration-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.046482 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046347    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac850265-e643-4365-b318-3507b8c838b9-hosts-file\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:55:37.046482 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046350    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-trusted-ca-bundle\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:37.046482 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046402    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfjxv\" (UniqueName: \"kubernetes.io/projected/ac850265-e643-4365-b318-3507b8c838b9-kube-api-access-tfjxv\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:55:37.046482 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046435    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.046482 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046463    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-script-lib\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046492    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-sys\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046543    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046587    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-socket-dir-parent\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046606    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-multus-certs\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046605    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27811319-29a6-459b-a1e3-8325c6cc2274-ovn-node-metrics-cert\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046646    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-lib-modules\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046670    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-multus\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046688    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-etc-kubernetes\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046718    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0213f1-3092-4a53-ba3a-1cec86701275-config-volume\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046734    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nflm\" (UniqueName: \"kubernetes.io/projected/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-kube-api-access-4nflm\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046750    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-netns\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046764    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-hostroot\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046785    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-conf-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.046837 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046810    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-kubernetes\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046921    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-systemd\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046944    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.046960    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0213f1-3092-4a53-ba3a-1cec86701275-config-volume\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047124    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-slash\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047142    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-k8s-cni-cncf-io\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047157    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-modprobe-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047173    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-data-volume\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047191    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-system-cni-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047219    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047280    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-run\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047304    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-data-volume\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047309    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-original-pull-secret\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047309    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cnibin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047345    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-bin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047370    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-cert\") pod \"ingress-canary-2d7p7\" (UID: \"2c7ca2bd-121d-4ef9-ae00-4a139497c91d\") " pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047396    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047413    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac850265-e643-4365-b318-3507b8c838b9-tmp-dir\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:55:37.047481 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047429    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-tuned\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047492    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac850265-e643-4365-b318-3507b8c838b9-tmp-dir\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047499    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047496    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-tuned\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047521    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-bin\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047553    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047578    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-cert\") pod \"ingress-canary-2d7p7\" (UID: \"2c7ca2bd-121d-4ef9-ae00-4a139497c91d\") " pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047579    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8vxl\" (UniqueName: \"kubernetes.io/projected/e18c7963-1218-4b83-837b-391b7d3edd22-kube-api-access-p8vxl\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047670    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047722    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-sys-fs\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047748    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-systemd-units\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047781    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-konnectivity-ca\") pod \"konnectivity-agent-m2tdr\" (UID: \"c8c74de0-5c52-4e2e-9f03-3b982a4e7667\") " pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047809    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-config\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047862    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cni-binary-copy\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.047986    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-kubelet\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048008    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-konnectivity-ca\") pod \"konnectivity-agent-m2tdr\" (UID: \"c8c74de0-5c52-4e2e-9f03-3b982a4e7667\") " pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048030    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-daemon-config\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048060    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-ovnkube-config\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048065    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2hwt\" (UniqueName: \"kubernetes.io/projected/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-kube-api-access-d2hwt\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048103    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-host\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:55:37.048155 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048128    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-device-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048169    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cni-binary-copy\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048173    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-daemon-config\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048197    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4tt\" (UniqueName: \"kubernetes.io/projected/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-kube-api-access-9m4tt\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048225    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048254    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-socket-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048279    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048305    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr48h\" (UniqueName: \"kubernetes.io/projected/2a73c32c-6071-4eb7-8215-171cb490ee56-kube-api-access-xr48h\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048345    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-tmp\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048406    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-etc-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048409    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1db5896f-bc1f-4965-9282-a344f5ad8e03-tmp\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048447    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-etc-selinux\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048478    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5txc\" (UniqueName: \"kubernetes.io/projected/1db5896f-bc1f-4965-9282-a344f5ad8e03-kube-api-access-b5txc\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048503    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-node-log\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048509    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-binary-copy\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048540    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-env-overrides\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048607    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048670    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysconfig\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048689    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-serviceca\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048712    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27811319-29a6-459b-a1e3-8325c6cc2274-env-overrides\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048741    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048770    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-iptables-alerter-script\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048835    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-agent-certs\") pod \"konnectivity-agent-m2tdr\" (UID: \"c8c74de0-5c52-4e2e-9f03-3b982a4e7667\") " pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048856    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-crio-socket\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048875    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbnn8\" (UniqueName: \"kubernetes.io/projected/a1df94c2-8be5-42df-9d04-7569e45f7648-kube-api-access-kbnn8\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048918    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-conf\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.048978 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.048977    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-iptables-alerter-script\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:55:37.049935 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.049025    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.049935 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.049122    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjv7m\" (UniqueName: \"kubernetes.io/projected/603f9058-eea9-4899-872c-503c7111dcc7-kube-api-access-tjv7m\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.049935 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.049169    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c8c74de0-5c52-4e2e-9f03-3b982a4e7667-agent-certs\") pod \"konnectivity-agent-m2tdr\" (UID: \"c8c74de0-5c52-4e2e-9f03-3b982a4e7667\") " pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:55:37.049935 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.049212    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9cqv\" (UniqueName: \"kubernetes.io/projected/32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65-kube-api-access-j9cqv\") pod \"migrator-66df5d496b-ctxf7\" (UID: \"32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7"
Oct 30 02:55:37.049935 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.049276    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-serviceca\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:55:37.049935 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.049300    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/603f9058-eea9-4899-872c-503c7111dcc7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.049935 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.049308    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e18c7963-1218-4b83-837b-391b7d3edd22-archives-path\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:37.049935 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.049356    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-cnibin\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.049935 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.049370    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e18c7963-1218-4b83-837b-391b7d3edd22-archives-path\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:37.049935 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.049382    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-kubelet-config\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:37.067374 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.067337    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxgw\" (UniqueName: \"kubernetes.io/projected/bf0213f1-3092-4a53-ba3a-1cec86701275-kube-api-access-ttxgw\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:37.067539 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.067476    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45bk\" (UniqueName: \"kubernetes.io/projected/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-kube-api-access-v45bk\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.067631 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.067603    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nflm\" (UniqueName: \"kubernetes.io/projected/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-kube-api-access-4nflm\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:55:37.067783 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.067760    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b86p\" (UniqueName: \"kubernetes.io/projected/27811319-29a6-459b-a1e3-8325c6cc2274-kube-api-access-2b86p\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.067933 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.067887    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c69r7\" (UniqueName: \"kubernetes.io/projected/2c7ca2bd-121d-4ef9-ae00-4a139497c91d-kube-api-access-c69r7\") pod \"ingress-canary-2d7p7\" (UID: \"2c7ca2bd-121d-4ef9-ae00-4a139497c91d\") " pod="openshift-ingress-canary/ingress-canary-2d7p7"
Oct 30 02:55:37.067987 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.067891    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfjxv\" (UniqueName: \"kubernetes.io/projected/ac850265-e643-4365-b318-3507b8c838b9-kube-api-access-tfjxv\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:55:37.068261 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.068246    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8vxl\" (UniqueName: \"kubernetes.io/projected/e18c7963-1218-4b83-837b-391b7d3edd22-kube-api-access-p8vxl\") pod \"periodic-gathering-clqt4-925xq\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") " pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:55:37.074647 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.074516    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2hwt\" (UniqueName: \"kubernetes.io/projected/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-kube-api-access-d2hwt\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.074647 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.074579    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbnn8\" (UniqueName: \"kubernetes.io/projected/a1df94c2-8be5-42df-9d04-7569e45f7648-kube-api-access-kbnn8\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.074647 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.074586    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjv7m\" (UniqueName: \"kubernetes.io/projected/603f9058-eea9-4899-872c-503c7111dcc7-kube-api-access-tjv7m\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.074647 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.074618    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9cqv\" (UniqueName: \"kubernetes.io/projected/32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65-kube-api-access-j9cqv\") pod \"migrator-66df5d496b-ctxf7\" (UID: \"32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-ctxf7"
Oct 30 02:55:37.074647 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.074650    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4tt\" (UniqueName: \"kubernetes.io/projected/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-kube-api-access-9m4tt\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:55:37.075033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.074690    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5txc\" (UniqueName: \"kubernetes.io/projected/1db5896f-bc1f-4965-9282-a344f5ad8e03-kube-api-access-b5txc\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.075033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.074757    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr48h\" (UniqueName: \"kubernetes.io/projected/2a73c32c-6071-4eb7-8215-171cb490ee56-kube-api-access-xr48h\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:37.075033 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.074877    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkz58\" (UniqueName: \"kubernetes.io/projected/23330a95-b468-4fa4-84d7-d5d58fbdf1cb-kube-api-access-lkz58\") pod \"network-check-target-rjwmx\" (UID: \"23330a95-b468-4fa4-84d7-d5d58fbdf1cb\") " pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:37.150086 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.149986    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.150086 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150035    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-netns\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150086 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150055    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-var-lib-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150086 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150072    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-host-slash\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:55:37.150086 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150088    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-var-lib-kubelet\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150104    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-log-socket\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150127    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-var-lib-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150145    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-log-socket\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150129    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-netns\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150167    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150198    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-netd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150196    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-var-lib-kubelet\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150142    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150145    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ca98f80-def1-4e19-8bc6-0f8327e2a5f1-host-slash\") pod \"iptables-alerter-6j76z\" (UID: \"8ca98f80-def1-4e19-8bc6-0f8327e2a5f1\") " pod="openshift-network-operator/iptables-alerter-6j76z"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150222    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-system-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150253    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-netd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150255    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-system-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150224    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-run-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150266    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-kubelet\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150290    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-systemd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150303    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-kubelet\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150308    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-ovn\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150329    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150343    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-systemd\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150377    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-registration-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150346    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-registration-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150382    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-ovn\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150408    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac850265-e643-4365-b318-3507b8c838b9-hosts-file\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150424    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.150426 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150438    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-sys\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150453    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150469    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-socket-dir-parent\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150439    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ac850265-e643-4365-b318-3507b8c838b9-hosts-file\") pod \"node-resolver-vsx5v\" (UID: \"ac850265-e643-4365-b318-3507b8c838b9\") " pod="openshift-dns/node-resolver-vsx5v"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150496    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-multus-certs\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150405    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-cni-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150521    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-run-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150513    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-lib-modules\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150547    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-multus-certs\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150523    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-sys\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150548    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150574    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-multus\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150576    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-socket-dir-parent\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150591    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-etc-kubernetes\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150609    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-netns\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150614    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-lib-modules\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150626    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-hostroot\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150646    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-conf-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150649    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-multus\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150655    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-etc-kubernetes\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150664    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-netns\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150681    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-multus-conf-dir\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150682    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-hostroot\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150697    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-kubernetes\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150712    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-systemd\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150741    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-slash\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150756    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-k8s-cni-cncf-io\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150761    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-kubernetes\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150771    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-modprobe-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150784    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-systemd\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150789    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-slash\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150790    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-system-cni-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150827    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-run-k8s-cni-cncf-io\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150854    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-system-cni-dir\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150880    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-run\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150934    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-run\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150943    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-modprobe-d\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150955    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cnibin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.150985    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-bin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151013    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151034    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-cni-bin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151037    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-bin\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151060    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-cni-bin\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151063    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151037    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-cnibin\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151065    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151095    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151124    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-sys-fs\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151103    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151142    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-systemd-units\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151171    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-sys-fs\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151172    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-systemd-units\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151184    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-kubelet\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151210    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-host\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151233    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-device-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:37.151228    4494 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151263    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cb55cf-0333-4d56-a6e0-4c13d0bd70a5-host-var-lib-kubelet\") pod \"multus-g5vjw\" (UID: \"92cb55cf-0333-4d56-a6e0-4c13d0bd70a5\") " pod="openshift-multus/multus-g5vjw"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151261    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151277    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33044703-7fec-4a2f-a1c7-cde0e8dd50bc-host\") pod \"node-ca-f8mgk\" (UID: \"33044703-7fec-4a2f-a1c7-cde0e8dd50bc\") " pod="openshift-image-registry/node-ca-f8mgk"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151291    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-socket-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151308    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-etc-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:37.151329    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls podName:bf0213f1-3092-4a53-ba3a-1cec86701275 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:37.651306006 +0000 UTC m=+1.857486221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls") pod "dns-default-zxwqk" (UID: "bf0213f1-3092-4a53-ba3a-1cec86701275") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151328    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-device-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151340    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-etc-openvswitch\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:37.151340    4494 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151356    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-etc-selinux\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151363    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-socket-dir\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151375    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-node-log\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:37.151396    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:37.651382913 +0000 UTC m=+1.857563110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151402    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27811319-29a6-459b-a1e3-8325c6cc2274-node-log\") pod \"ovnkube-node-xltm9\" (UID: \"27811319-29a6-459b-a1e3-8325c6cc2274\") " pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151421    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151431    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1df94c2-8be5-42df-9d04-7569e45f7648-etc-selinux\") pod \"aws-ebs-csi-driver-node-zrg6t\" (UID: \"a1df94c2-8be5-42df-9d04-7569e45f7648\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrg6t"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151436    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysconfig\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151466    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysconfig\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151476    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-crio-socket\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151499    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-conf\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151525    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-cnibin\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151546    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-kubelet-config\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:37.151477 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151549    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-crio-socket\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.153729 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151565    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-host\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.153729 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151573    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/603f9058-eea9-4899-872c-503c7111dcc7-cnibin\") pod \"multus-additional-cni-plugins-cspkf\" (UID: \"603f9058-eea9-4899-872c-503c7111dcc7\") " pod="openshift-multus/multus-additional-cni-plugins-cspkf"
Oct 30 02:55:37.153729 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151588    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-dbus\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:37.153729 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151596    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-kubelet-config\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:37.153729 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151622    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-host\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.153729 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151634    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1db5896f-bc1f-4965-9282-a344f5ad8e03-etc-sysctl-conf\") pod \"tuned-p6dt8\" (UID: \"1db5896f-bc1f-4965-9282-a344f5ad8e03\") " pod="openshift-cluster-node-tuning-operator/tuned-p6dt8"
Oct 30 02:55:37.153729 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151689    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d303f31-40f2-46fc-9bb1-73ba2851a5ec-dbus\") pod \"global-pull-secret-syncer-ddlpl\" (UID: \"0d303f31-40f2-46fc-9bb1-73ba2851a5ec\") " pod="kube-system/global-pull-secret-syncer-ddlpl"
Oct 30 02:55:37.153729 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.151780    4494 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:55:37.154060 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.154044    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a73c32c-6071-4eb7-8215-171cb490ee56-metrics-certs\") pod \"network-metrics-daemon-ndbtf\" (UID: \"2a73c32c-6071-4eb7-8215-171cb490ee56\") " pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:37.238626 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.238595    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndbtf"
Oct 30 02:55:37.294300 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.294276    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-ctxf7_32a2baa7-1ca5-4dd5-ba93-44e84e3e6f65/migrator/0.log"
Oct 30 02:55:37.559788 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.559764    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ndbtf"]
Oct 30 02:55:37.561681 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:37.561638    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a73c32c_6071_4eb7_8215_171cb490ee56.slice/crio-870faefccc8dd4195b475662b5b63a7bca58668ec347d6eb1ece3d662dee4fb3 WatchSource:0}: Error finding container 870faefccc8dd4195b475662b5b63a7bca58668ec347d6eb1ece3d662dee4fb3: Status 404 returned error can't find the container with id 870faefccc8dd4195b475662b5b63a7bca58668ec347d6eb1ece3d662dee4fb3
Oct 30 02:55:37.564090 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.564068    4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:55:37.654694 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.654655    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:37.654694 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.654700    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:37.654965 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:37.654822    4494 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:37.654965 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:37.654822    4494 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:37.654965 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:37.654945    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls podName:bf0213f1-3092-4a53-ba3a-1cec86701275 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:38.654925402 +0000 UTC m=+2.861105616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls") pod "dns-default-zxwqk" (UID: "bf0213f1-3092-4a53-ba3a-1cec86701275") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:37.654965 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:37.654962    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:38.654956018 +0000 UTC m=+2.861136214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:37.970171 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:37.970075    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ndbtf" event={"ID":"2a73c32c-6071-4eb7-8215-171cb490ee56","Type":"ContainerStarted","Data":"870faefccc8dd4195b475662b5b63a7bca58668ec347d6eb1ece3d662dee4fb3"}
Oct 30 02:55:38.376246 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:38.376206    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:55:38.478385 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:38.478355    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:38.663372 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:38.663287    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:38.663372 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:38.663334    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:38.663593 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:38.663461    4494 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:38.663593 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:38.663462    4494 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:38.663593 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:38.663522    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:40.663507322 +0000 UTC m=+4.869687540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:38.663593 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:38.663537    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls podName:bf0213f1-3092-4a53-ba3a-1cec86701275 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:40.663531344 +0000 UTC m=+4.869711540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls") pod "dns-default-zxwqk" (UID: "bf0213f1-3092-4a53-ba3a-1cec86701275") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:38.664026 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:38.664005    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:38.972649 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:38.972574    4494 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:39.977192 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:39.977150    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ndbtf" event={"ID":"2a73c32c-6071-4eb7-8215-171cb490ee56","Type":"ContainerStarted","Data":"afea0655c787db0c0d6601cd42c42cd8eaec96347208b222aed45d96a4d328ea"}
Oct 30 02:55:39.977583 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:39.977207    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ndbtf" event={"ID":"2a73c32c-6071-4eb7-8215-171cb490ee56","Type":"ContainerStarted","Data":"5ab823b96ac2ad7d3fbcd51cb725876963bdc63f4920c9cbd1cf6810ddf413c0"}
Oct 30 02:55:40.678683 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:40.678634    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:40.678683 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:40.678686    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:40.678997 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:40.678792    4494 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:40.678997 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:40.678795    4494 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:40.678997 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:40.678847    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls podName:f63a54a5-8e93-4f23-a7da-a3a22d63f2b2 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:44.678833922 +0000 UTC m=+8.885014131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgwbh" (UID: "f63a54a5-8e93-4f23-a7da-a3a22d63f2b2") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:40.678997 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:40.678883    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls podName:bf0213f1-3092-4a53-ba3a-1cec86701275 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:44.678856489 +0000 UTC m=+8.885036704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls") pod "dns-default-zxwqk" (UID: "bf0213f1-3092-4a53-ba3a-1cec86701275") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:41.429126 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:41.429094    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m2tdr"
Oct 30 02:55:43.918277 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:43.918248    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xltm9"
Oct 30 02:55:44.706470 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:44.706425    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:44.706470 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:44.706473    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:44.709064 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:44.709032    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f63a54a5-8e93-4f23-a7da-a3a22d63f2b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgwbh\" (UID: \"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2\") " pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:44.709187 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:44.709032    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf0213f1-3092-4a53-ba3a-1cec86701275-metrics-tls\") pod \"dns-default-zxwqk\" (UID: \"bf0213f1-3092-4a53-ba3a-1cec86701275\") " pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:44.741157 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:44.741122    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgwbh"
Oct 30 02:55:44.741325 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:44.741200    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:45.063725 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:45.063699    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zxwqk"]
Oct 30 02:55:45.065408 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:45.065376    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0213f1_3092_4a53_ba3a_1cec86701275.slice/crio-9e10fddea2b12df42de31741976ce8f4ddbb9fff1050743b3382b8c0369e7497 WatchSource:0}: Error finding container 9e10fddea2b12df42de31741976ce8f4ddbb9fff1050743b3382b8c0369e7497: Status 404 returned error can't find the container with id 9e10fddea2b12df42de31741976ce8f4ddbb9fff1050743b3382b8c0369e7497
Oct 30 02:55:45.075447 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:45.075414    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63a54a5_8e93_4f23_a7da_a3a22d63f2b2.slice/crio-ae2f3292006e5b2c07c9575bb1651ee33d0121d5b96fe398574b0b09dd0c857e WatchSource:0}: Error finding container ae2f3292006e5b2c07c9575bb1651ee33d0121d5b96fe398574b0b09dd0c857e: Status 404 returned error can't find the container with id ae2f3292006e5b2c07c9575bb1651ee33d0121d5b96fe398574b0b09dd0c857e
Oct 30 02:55:45.076552 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:45.076518    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pgwbh"]
Oct 30 02:55:45.993728 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:45.993624    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zxwqk" event={"ID":"bf0213f1-3092-4a53-ba3a-1cec86701275","Type":"ContainerStarted","Data":"9e10fddea2b12df42de31741976ce8f4ddbb9fff1050743b3382b8c0369e7497"}
Oct 30 02:55:45.995922 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:45.995873    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgwbh" event={"ID":"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2","Type":"ContainerStarted","Data":"259024a0996bf54472318aa18a489a6ca844049f6f6464b92137a39bba0a3a46"}
Oct 30 02:55:45.996097 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:45.995960    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgwbh" event={"ID":"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2","Type":"ContainerStarted","Data":"5751e3f09e9926a1b273762a8a1f398d75cc8dfea63afd45625dfeb4b160e78e"}
Oct 30 02:55:45.996097 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:45.995992    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgwbh" event={"ID":"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2","Type":"ContainerStarted","Data":"ae2f3292006e5b2c07c9575bb1651ee33d0121d5b96fe398574b0b09dd0c857e"}
Oct 30 02:55:47.006338 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:47.006254    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zxwqk" event={"ID":"bf0213f1-3092-4a53-ba3a-1cec86701275","Type":"ContainerStarted","Data":"1854a7a008bd1b59233b9da5703164debc9d5cc89d13c41cf1b42e9de4e112dd"}
Oct 30 02:55:47.006815 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:47.006350    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zxwqk" event={"ID":"bf0213f1-3092-4a53-ba3a-1cec86701275","Type":"ContainerStarted","Data":"419197a9cb7d94f83dd08b540d97b96408fbbe2dbe2e34037b8641d2a48450ed"}
Oct 30 02:55:47.006815 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:47.006399    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:48.012126 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:48.012083    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgwbh" event={"ID":"f63a54a5-8e93-4f23-a7da-a3a22d63f2b2","Type":"ContainerStarted","Data":"9715be4a0a497ef468c086ec11165d4f72618e97cc6239747b8f138f0c261eed"}
Oct 30 02:55:49.737090 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.737044    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-dvbg6"]
Oct 30 02:55:49.741820 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.741792    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-dvbg6"
Oct 30 02:55:49.753335 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.752992    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-567496975-sqqjl"]
Oct 30 02:55:49.756397 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.756375    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.761423 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.761226    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-hjxm5\""
Oct 30 02:55:49.761423 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.761274    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 02:55:49.761423 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.761396    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 02:55:49.771683 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.771650    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567496975-sqqjl"]
Oct 30 02:55:49.772730 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.772257    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-dvbg6"]
Oct 30 02:55:49.775570 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.775541    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 02:55:49.775681 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.775643    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 02:55:49.775739 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.775723    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 02:55:49.838843 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.838809    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c22sp\" (UniqueName: \"kubernetes.io/projected/c2c612b6-be82-4c17-a26b-06c6768cae0f-kube-api-access-c22sp\") pod \"downloads-6dd789c8b7-dvbg6\" (UID: \"c2c612b6-be82-4c17-a26b-06c6768cae0f\") " pod="openshift-console/downloads-6dd789c8b7-dvbg6"
Oct 30 02:55:49.838843 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.838854    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b7918c-ed94-4a74-9f83-e6cb251a5885-tmp\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.839106 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.838920    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2c612b6-be82-4c17-a26b-06c6768cae0f-tmp\") pod \"downloads-6dd789c8b7-dvbg6\" (UID: \"c2c612b6-be82-4c17-a26b-06c6768cae0f\") " pod="openshift-console/downloads-6dd789c8b7-dvbg6"
Oct 30 02:55:49.839106 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.839053    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-config\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.839106 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.839085    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh9j8\" (UniqueName: \"kubernetes.io/projected/61b7918c-ed94-4a74-9f83-e6cb251a5885-kube-api-access-rh9j8\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.839218 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.839114    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-serving-cert\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.839218 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.839151    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-service-ca\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.839300 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.839205    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-oauth-config\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.839300 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.839247    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-oauth-serving-cert\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.865108 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.865065    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 02:55:49.865301 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.865141    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 02:55:49.865301 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.865224    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pxwqq\""
Oct 30 02:55:49.940051 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940005    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-oauth-config\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.940051 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940048    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-oauth-serving-cert\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.940313 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940079    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c22sp\" (UniqueName: \"kubernetes.io/projected/c2c612b6-be82-4c17-a26b-06c6768cae0f-kube-api-access-c22sp\") pod \"downloads-6dd789c8b7-dvbg6\" (UID: \"c2c612b6-be82-4c17-a26b-06c6768cae0f\") " pod="openshift-console/downloads-6dd789c8b7-dvbg6"
Oct 30 02:55:49.940313 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940107    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b7918c-ed94-4a74-9f83-e6cb251a5885-tmp\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.940313 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940136    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2c612b6-be82-4c17-a26b-06c6768cae0f-tmp\") pod \"downloads-6dd789c8b7-dvbg6\" (UID: \"c2c612b6-be82-4c17-a26b-06c6768cae0f\") " pod="openshift-console/downloads-6dd789c8b7-dvbg6"
Oct 30 02:55:49.940313 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940193    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-config\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.940313 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940213    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh9j8\" (UniqueName: \"kubernetes.io/projected/61b7918c-ed94-4a74-9f83-e6cb251a5885-kube-api-access-rh9j8\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.940313 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940239    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-serving-cert\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.940313 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940267    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-service-ca\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.940688 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940661    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2c612b6-be82-4c17-a26b-06c6768cae0f-tmp\") pod \"downloads-6dd789c8b7-dvbg6\" (UID: \"c2c612b6-be82-4c17-a26b-06c6768cae0f\") " pod="openshift-console/downloads-6dd789c8b7-dvbg6"
Oct 30 02:55:49.940760 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.940686    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b7918c-ed94-4a74-9f83-e6cb251a5885-tmp\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.941027 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.941003    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-config\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.941146 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.941108    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-oauth-serving-cert\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.941254 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.941235    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-service-ca\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.942743 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.942717    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-oauth-config\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.942865 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.942848    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-serving-cert\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.964535 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.964507    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh9j8\" (UniqueName: \"kubernetes.io/projected/61b7918c-ed94-4a74-9f83-e6cb251a5885-kube-api-access-rh9j8\") pod \"console-567496975-sqqjl\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") " pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:49.964673 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:49.964605    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c22sp\" (UniqueName: \"kubernetes.io/projected/c2c612b6-be82-4c17-a26b-06c6768cae0f-kube-api-access-c22sp\") pod \"downloads-6dd789c8b7-dvbg6\" (UID: \"c2c612b6-be82-4c17-a26b-06c6768cae0f\") " pod="openshift-console/downloads-6dd789c8b7-dvbg6"
Oct 30 02:55:50.061431 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:50.061397    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-dvbg6"
Oct 30 02:55:50.070934 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:50.070892    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:55:50.381283 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:50.381259    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-dvbg6"]
Oct 30 02:55:50.382805 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:50.382777    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c612b6_be82_4c17_a26b_06c6768cae0f.slice/crio-6151eeb0b033ecc64dac4aeb199d5debfb568d9aae8d33eeee6283914ea0783e WatchSource:0}: Error finding container 6151eeb0b033ecc64dac4aeb199d5debfb568d9aae8d33eeee6283914ea0783e: Status 404 returned error can't find the container with id 6151eeb0b033ecc64dac4aeb199d5debfb568d9aae8d33eeee6283914ea0783e
Oct 30 02:55:50.384281 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:50.384256    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567496975-sqqjl"]
Oct 30 02:55:50.393658 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:50.393630    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b7918c_ed94_4a74_9f83_e6cb251a5885.slice/crio-1e371d05cbc10b25dbdc85c1b4efb9698d53cdc09656fd6c1e216ce1b4a69ff6 WatchSource:0}: Error finding container 1e371d05cbc10b25dbdc85c1b4efb9698d53cdc09656fd6c1e216ce1b4a69ff6: Status 404 returned error can't find the container with id 1e371d05cbc10b25dbdc85c1b4efb9698d53cdc09656fd6c1e216ce1b4a69ff6
Oct 30 02:55:50.858020 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:50.857979    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk"]
Oct 30 02:55:50.862168 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:50.861369    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk"
Oct 30 02:55:50.879411 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:50.879384    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ghknm\""
Oct 30 02:55:50.881715 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:50.881662    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 02:55:50.885158 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:50.882922    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk"]
Oct 30 02:55:50.951089 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:50.951046    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/525f461c-2548-41c0-b83c-6bad98da93ad-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bw5fk\" (UID: \"525f461c-2548-41c0-b83c-6bad98da93ad\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk"
Oct 30 02:55:51.025234 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:51.025176    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-dvbg6" event={"ID":"c2c612b6-be82-4c17-a26b-06c6768cae0f","Type":"ContainerStarted","Data":"6151eeb0b033ecc64dac4aeb199d5debfb568d9aae8d33eeee6283914ea0783e"}
Oct 30 02:55:51.027377 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:51.027351    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567496975-sqqjl" event={"ID":"61b7918c-ed94-4a74-9f83-e6cb251a5885","Type":"ContainerStarted","Data":"1e371d05cbc10b25dbdc85c1b4efb9698d53cdc09656fd6c1e216ce1b4a69ff6"}
Oct 30 02:55:51.052159 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:51.052119    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/525f461c-2548-41c0-b83c-6bad98da93ad-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bw5fk\" (UID: \"525f461c-2548-41c0-b83c-6bad98da93ad\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk"
Oct 30 02:55:51.055455 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:51.055422    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/525f461c-2548-41c0-b83c-6bad98da93ad-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bw5fk\" (UID: \"525f461c-2548-41c0-b83c-6bad98da93ad\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk"
Oct 30 02:55:51.178320 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:51.178019    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk"
Oct 30 02:55:51.488828 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:51.488775    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk"]
Oct 30 02:55:51.492093 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:51.492060    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525f461c_2548_41c0_b83c_6bad98da93ad.slice/crio-f089601779a26b9a59a22353ac7e4715d1a9bbffa7fb1f2b4bee251c0ea92de9 WatchSource:0}: Error finding container f089601779a26b9a59a22353ac7e4715d1a9bbffa7fb1f2b4bee251c0ea92de9: Status 404 returned error can't find the container with id f089601779a26b9a59a22353ac7e4715d1a9bbffa7fb1f2b4bee251c0ea92de9
Oct 30 02:55:52.032236 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:52.032196    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk" event={"ID":"525f461c-2548-41c0-b83c-6bad98da93ad","Type":"ContainerStarted","Data":"f089601779a26b9a59a22353ac7e4715d1a9bbffa7fb1f2b4bee251c0ea92de9"}
Oct 30 02:55:53.038987 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:53.038947    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk" event={"ID":"525f461c-2548-41c0-b83c-6bad98da93ad","Type":"ContainerStarted","Data":"392ed5515df4c5259a1b25f00c3551ef3b1cd2678d9fd59354177b8caf06b5c7"}
Oct 30 02:55:53.040023 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:53.039961    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk"
Oct 30 02:55:53.045492 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:53.045438    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk"
Oct 30 02:55:53.081791 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:53.081717    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bw5fk" podStartSLOduration=1.97580505 podStartE2EDuration="3.081701717s" totalImagesPullingTime="1.105896667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:53.081297968 +0000 UTC m=+17.287478209" watchObservedRunningTime="2026-10-30 02:55:53.081701717 +0000 UTC m=+17.287881931"
Oct 30 02:55:54.046505 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:54.046455    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567496975-sqqjl" event={"ID":"61b7918c-ed94-4a74-9f83-e6cb251a5885","Type":"ContainerStarted","Data":"e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec"}
Oct 30 02:55:54.087174 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:55:54.087133    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 02:55:54.095788 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:54.095733    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-567496975-sqqjl" podStartSLOduration=2.054931404 podStartE2EDuration="5.095718484s" totalImagesPullingTime="3.04078708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:54.095200669 +0000 UTC m=+18.301380887" watchObservedRunningTime="2026-10-30 02:55:54.095718484 +0000 UTC m=+18.301898698"
Oct 30 02:55:57.014999 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:57.014954    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zxwqk"
Oct 30 02:55:58.632354 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.632314    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"]
Oct 30 02:55:58.636235 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.636202    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.657267 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.657237    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 02:55:58.666528 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.666461    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:55:58.666717 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.666699    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 02:55:58.666777 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.666731    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:55:58.669504 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.669456    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 02:55:58.669504 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.669496    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-b4xxn\""
Oct 30 02:55:58.670974 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.670948    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"]
Oct 30 02:55:58.681744 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.681711    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sfkc2"]
Oct 30 02:55:58.685017 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.684980    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"]
Oct 30 02:55:58.685185 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.685129    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.688756 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.688730    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.707717 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.707684    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"]
Oct 30 02:55:58.724656 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.724629    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 02:55:58.724858 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.724775    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 02:55:58.724858 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.724812    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-szgzh\""
Oct 30 02:55:58.724858 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.724845    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 02:55:58.724858 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.724850    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 02:55:58.725052 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.724892    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-plxxd\""
Oct 30 02:55:58.757570 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.757541    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 02:55:58.758010 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.757989    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 02:55:58.806622 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806588    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-tls\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.806622 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806625    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8386b555-b07c-4481-9226-4a65739fd8db-sys\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.806876 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806653    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf9aaba6-e684-4534-bbf9-4396fc97797a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.806876 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806675    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtxc\" (UniqueName: \"kubernetes.io/projected/bf9aaba6-e684-4534-bbf9-4396fc97797a-kube-api-access-mvtxc\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.806876 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806697    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-accelerators-collector-config\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.806876 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806717    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-wtmp\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.806876 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806744    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8386b555-b07c-4481-9226-4a65739fd8db-metrics-client-ca\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.806876 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806778    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg92d\" (UniqueName: \"kubernetes.io/projected/8386b555-b07c-4481-9226-4a65739fd8db-kube-api-access-vg92d\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.806876 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806797    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf9aaba6-e684-4534-bbf9-4396fc97797a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.806876 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806820    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf9aaba6-e684-4534-bbf9-4396fc97797a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.806876 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806849    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.806876 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806869    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41def15-5a16-4b6d-b79a-362c08010b6b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.807269 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806894    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.807269 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806947    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e41def15-5a16-4b6d-b79a-362c08010b6b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.807269 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806965    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.807269 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.806988    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-textfile\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.807269 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.807019    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.807269 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.807036    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfv9z\" (UniqueName: \"kubernetes.io/projected/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-api-access-bfv9z\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.807269 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.807073    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8386b555-b07c-4481-9226-4a65739fd8db-root\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.908581 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908486    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg92d\" (UniqueName: \"kubernetes.io/projected/8386b555-b07c-4481-9226-4a65739fd8db-kube-api-access-vg92d\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.908581 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908533    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf9aaba6-e684-4534-bbf9-4396fc97797a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.908581 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908555    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf9aaba6-e684-4534-bbf9-4396fc97797a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.908850 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908588    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.908850 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908620    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41def15-5a16-4b6d-b79a-362c08010b6b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.908850 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908648    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.908850 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908684    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e41def15-5a16-4b6d-b79a-362c08010b6b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.908850 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908706    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.908850 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908733    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-textfile\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.908850 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908768    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.908850 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908792    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfv9z\" (UniqueName: \"kubernetes.io/projected/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-api-access-bfv9z\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.908850 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908836    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8386b555-b07c-4481-9226-4a65739fd8db-root\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.909270 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908867    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-tls\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.909270 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908891    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8386b555-b07c-4481-9226-4a65739fd8db-sys\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.909270 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908935    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf9aaba6-e684-4534-bbf9-4396fc97797a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.909270 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908961    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtxc\" (UniqueName: \"kubernetes.io/projected/bf9aaba6-e684-4534-bbf9-4396fc97797a-kube-api-access-mvtxc\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.909270 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.908991    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-accelerators-collector-config\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.909270 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.909018    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-wtmp\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.909270 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.909050    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8386b555-b07c-4481-9226-4a65739fd8db-metrics-client-ca\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.909549 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.909451    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8386b555-b07c-4481-9226-4a65739fd8db-sys\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.910569 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.909694    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8386b555-b07c-4481-9226-4a65739fd8db-metrics-client-ca\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.910569 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.909759    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8386b555-b07c-4481-9226-4a65739fd8db-root\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.910569 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.909828    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.910569 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.910116    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-textfile\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.910569 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.910139    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf9aaba6-e684-4534-bbf9-4396fc97797a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.910569 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.910279    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-wtmp\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.910569 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.910456    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e41def15-5a16-4b6d-b79a-362c08010b6b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.910569 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.910532    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e41def15-5a16-4b6d-b79a-362c08010b6b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.910958 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.910783    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-accelerators-collector-config\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.912774 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.912749    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.912980 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.912956    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf9aaba6-e684-4534-bbf9-4396fc97797a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.913250 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.913232    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.913856 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.913834    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8386b555-b07c-4481-9226-4a65739fd8db-node-exporter-tls\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.914129 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.914106    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.916131 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.916109    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf9aaba6-e684-4534-bbf9-4396fc97797a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.933581 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.933517    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg92d\" (UniqueName: \"kubernetes.io/projected/8386b555-b07c-4481-9226-4a65739fd8db-kube-api-access-vg92d\") pod \"node-exporter-sfkc2\" (UID: \"8386b555-b07c-4481-9226-4a65739fd8db\") " pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:58.935389 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.935362    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfv9z\" (UniqueName: \"kubernetes.io/projected/e41def15-5a16-4b6d-b79a-362c08010b6b-kube-api-access-bfv9z\") pod \"kube-state-metrics-5dc8d7d89b-z74j4\" (UID: \"e41def15-5a16-4b6d-b79a-362c08010b6b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:58.935537 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.935444    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtxc\" (UniqueName: \"kubernetes.io/projected/bf9aaba6-e684-4534-bbf9-4396fc97797a-kube-api-access-mvtxc\") pod \"openshift-state-metrics-7bd7489957-dw86r\" (UID: \"bf9aaba6-e684-4534-bbf9-4396fc97797a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:58.955123 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:58.955088    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"
Oct 30 02:55:59.004683 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.004640    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sfkc2"
Oct 30 02:55:59.016954 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.016691    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"
Oct 30 02:55:59.065403 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.065334    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sfkc2" event={"ID":"8386b555-b07c-4481-9226-4a65739fd8db","Type":"ContainerStarted","Data":"ad911c874e2c7686d10c25c5e82c6e6842771d572c1622eda083ca3bbf84e58c"}
Oct 30 02:55:59.269716 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.269685    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r"]
Oct 30 02:55:59.271751 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:59.271720    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9aaba6_e684_4534_bbf9_4396fc97797a.slice/crio-c5cfd92b4c2be7f464d6c1bafdbe8d28e79b8897155715e15a3bcfef150da848 WatchSource:0}: Error finding container c5cfd92b4c2be7f464d6c1bafdbe8d28e79b8897155715e15a3bcfef150da848: Status 404 returned error can't find the container with id c5cfd92b4c2be7f464d6c1bafdbe8d28e79b8897155715e15a3bcfef150da848
Oct 30 02:55:59.331196 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.331033    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4"]
Oct 30 02:55:59.341347 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:55:59.334627    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41def15_5a16_4b6d_b79a_362c08010b6b.slice/crio-c471fdac7bbb2966be2eec25595261e210fce4851e042cc171888cf01cdd2445 WatchSource:0}: Error finding container c471fdac7bbb2966be2eec25595261e210fce4851e042cc171888cf01cdd2445: Status 404 returned error can't find the container with id c471fdac7bbb2966be2eec25595261e210fce4851e042cc171888cf01cdd2445
Oct 30 02:55:59.664865 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.664819    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-866cf6d967-4jkj7"]
Oct 30 02:55:59.685328 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.685287    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.707388 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.707350    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866cf6d967-4jkj7"]
Oct 30 02:55:59.720740 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.720709    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 02:55:59.816954 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.816880    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-console-config\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.817133 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.817025    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnkt\" (UniqueName: \"kubernetes.io/projected/1b582873-8125-4822-aa57-4a387907ac78-kube-api-access-6dnkt\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.817133 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.817072    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-trusted-ca-bundle\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.817133 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.817106    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-serving-cert\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.817133 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.817122    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-oauth-config\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.817332 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.817140    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-service-ca\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.817332 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.817156    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-oauth-serving-cert\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.817332 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.817191    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b582873-8125-4822-aa57-4a387907ac78-tmp\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.917938 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.917834    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnkt\" (UniqueName: \"kubernetes.io/projected/1b582873-8125-4822-aa57-4a387907ac78-kube-api-access-6dnkt\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.917938 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.917892    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-trusted-ca-bundle\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.918140 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.917954    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-serving-cert\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.918140 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.917976    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-oauth-config\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.918140 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.918003    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-service-ca\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.918140 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.918025    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-oauth-serving-cert\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.918140 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.918055    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b582873-8125-4822-aa57-4a387907ac78-tmp\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.918140 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.918123    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-console-config\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.918976 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.918950    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-console-config\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.920090 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.920065    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-trusted-ca-bundle\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.920616 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.920593    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-oauth-serving-cert\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.920616 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.920604    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-service-ca\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.920967 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.920892    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b582873-8125-4822-aa57-4a387907ac78-tmp\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.923086 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.923061    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-serving-cert\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.923401 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.923380    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-oauth-config\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:55:59.946400 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:55:59.946369    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnkt\" (UniqueName: \"kubernetes.io/projected/1b582873-8125-4822-aa57-4a387907ac78-kube-api-access-6dnkt\") pod \"console-866cf6d967-4jkj7\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") " pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:56:00.022620 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.022338    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:56:00.071420 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.071384    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:56:00.072150 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.071818    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:56:00.075165 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.075121    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r" event={"ID":"bf9aaba6-e684-4534-bbf9-4396fc97797a","Type":"ContainerStarted","Data":"6da90704aded17624a88e62e2e1fd22fbfd85637429b18d2177f87bba2400d0e"}
Oct 30 02:56:00.075320 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.075207    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r" event={"ID":"bf9aaba6-e684-4534-bbf9-4396fc97797a","Type":"ContainerStarted","Data":"68d168fa5989fda4579031a2ccddc83c9f3f1f0727c328e938d86c6921b03857"}
Oct 30 02:56:00.075320 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.075259    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r" event={"ID":"bf9aaba6-e684-4534-bbf9-4396fc97797a","Type":"ContainerStarted","Data":"c5cfd92b4c2be7f464d6c1bafdbe8d28e79b8897155715e15a3bcfef150da848"}
Oct 30 02:56:00.079255 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.078701    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:56:00.080470 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.080440    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4" event={"ID":"e41def15-5a16-4b6d-b79a-362c08010b6b","Type":"ContainerStarted","Data":"c471fdac7bbb2966be2eec25595261e210fce4851e042cc171888cf01cdd2445"}
Oct 30 02:56:00.178994 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.176340    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:56:00.200566 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.200517    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.214375 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.213308    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:56:00.216612 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.216568    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 02:56:00.218951 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.218887    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xphk4\""
Oct 30 02:56:00.223342 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.223290    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 02:56:00.317271 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.317004    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 02:56:00.317271 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.317088    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 02:56:00.317271 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.317004    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 02:56:00.317271 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.317145    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 02:56:00.317271 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.317218    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 02:56:00.317271 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.317219    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 02:56:00.317643 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.317295    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 02:56:00.322776 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.322737    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.322959 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.322798    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.322959 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.322911    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.323049 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.322959    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.323049 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.322996    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.323109 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.323056    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-config-out\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.323109 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.323087    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.323166 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.323130    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-tls-assets\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.323166 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.323145    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmq6h\" (UniqueName: \"kubernetes.io/projected/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-kube-api-access-vmq6h\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.323223 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.323171    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-web-config\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.323259 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.323247    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.323307 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.323300    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-config-volume\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.323363 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.323314    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.349543 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.349516    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866cf6d967-4jkj7"]
Oct 30 02:56:00.415610 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:56:00.415572    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b582873_8125_4822_aa57_4a387907ac78.slice/crio-5bfc34fc0617ea4fb7174b2586cd6169d69dbccdac80edbcfb3a6230f5b0580b WatchSource:0}: Error finding container 5bfc34fc0617ea4fb7174b2586cd6169d69dbccdac80edbcfb3a6230f5b0580b: Status 404 returned error can't find the container with id 5bfc34fc0617ea4fb7174b2586cd6169d69dbccdac80edbcfb3a6230f5b0580b
Oct 30 02:56:00.424424 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424390    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-web-config\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424580 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424451    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424580 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424478    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-config-volume\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424580 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424493    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424580 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424540    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424580 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424572    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424874 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424598    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424874 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424616    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424874 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424648    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424874 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424681    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-config-out\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424874 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424699    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424874 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424724    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-tls-assets\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.424874 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.424741    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmq6h\" (UniqueName: \"kubernetes.io/projected/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-kube-api-access-vmq6h\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.425442 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.425410    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.425528 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.425437    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.426612 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.425982    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.427692 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.427643    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-config-out\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.429131 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.429049    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-tls-assets\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.429884 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.429840    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-web-config\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.430200 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.430180    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.430868 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.430818    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.431686 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.431663    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.432233 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.432190    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-config-volume\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.432320 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.432267    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.432656 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.432632    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.450012 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.449982    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmq6h\" (UniqueName: \"kubernetes.io/projected/52ce43e0-2ca8-4559-9fcb-3ddad3b9e169-kube-api-access-vmq6h\") pod \"alertmanager-main-0\" (UID: \"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.597026 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.596984    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 02:56:00.940860 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.940769    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7595945dc8-drcrc"]
Oct 30 02:56:00.957859 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.957816    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:00.964406 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.964376    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 02:56:00.966192 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:56:00.966159    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ce43e0_2ca8_4559_9fcb_3ddad3b9e169.slice/crio-1dc8142285fb448b45e1983350c8cb1b545c4e0951e9dfbe60df7ce68cc156c9 WatchSource:0}: Error finding container 1dc8142285fb448b45e1983350c8cb1b545c4e0951e9dfbe60df7ce68cc156c9: Status 404 returned error can't find the container with id 1dc8142285fb448b45e1983350c8cb1b545c4e0951e9dfbe60df7ce68cc156c9
Oct 30 02:56:00.974313 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.974190    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 02:56:00.974313 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.974253    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 02:56:00.974508 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.974431    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 02:56:00.974812 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.974695    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tqfhn\""
Oct 30 02:56:00.974812 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.974761    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7595945dc8-drcrc"]
Oct 30 02:56:00.974812 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.974779    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5b0isi3u5qgll\""
Oct 30 02:56:00.974812 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.974799    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 02:56:00.977983 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:00.977962    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 02:56:01.091087 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.091042    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866cf6d967-4jkj7" event={"ID":"1b582873-8125-4822-aa57-4a387907ac78","Type":"ContainerStarted","Data":"ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac"}
Oct 30 02:56:01.091275 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.091105    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866cf6d967-4jkj7" event={"ID":"1b582873-8125-4822-aa57-4a387907ac78","Type":"ContainerStarted","Data":"5bfc34fc0617ea4fb7174b2586cd6169d69dbccdac80edbcfb3a6230f5b0580b"}
Oct 30 02:56:01.092953 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.092922    4494 generic.go:356] "Generic (PLEG): container finished" podID="8386b555-b07c-4481-9226-4a65739fd8db" containerID="9a862b2dab71da69ce5be611dea764cd30625593048588cf7b32222766f3c2d2" exitCode=0
Oct 30 02:56:01.093078 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.092961    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sfkc2" event={"ID":"8386b555-b07c-4481-9226-4a65739fd8db","Type":"ContainerDied","Data":"9a862b2dab71da69ce5be611dea764cd30625593048588cf7b32222766f3c2d2"}
Oct 30 02:56:01.094449 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.094426    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169","Type":"ContainerStarted","Data":"1dc8142285fb448b45e1983350c8cb1b545c4e0951e9dfbe60df7ce68cc156c9"}
Oct 30 02:56:01.098921 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.098884    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:56:01.131375 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.131328    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.131375 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.131384    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.131621 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.131404    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.131621 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.131541    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-tls\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.131621 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.131597    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7f7009b-d19b-43ea-a7d1-1901c90daf79-metrics-client-ca\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.131621 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.131618    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.131817 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.131636    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-grpc-tls\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.131817 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.131679    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7f7009b-d19b-43ea-a7d1-1901c90daf79-tmp\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.131817 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.131705    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpggm\" (UniqueName: \"kubernetes.io/projected/d7f7009b-d19b-43ea-a7d1-1901c90daf79-kube-api-access-wpggm\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.134344 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.134297    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-866cf6d967-4jkj7" podStartSLOduration=2.134279538 podStartE2EDuration="2.134279538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:01.133733426 +0000 UTC m=+25.339913643" watchObservedRunningTime="2026-10-30 02:56:01.134279538 +0000 UTC m=+25.340459750"
Oct 30 02:56:01.234103 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.232351    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.234103 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.232431    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.234103 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.232460    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.234103 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.232523    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-tls\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.234103 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.232555    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7f7009b-d19b-43ea-a7d1-1901c90daf79-metrics-client-ca\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.234103 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.232581    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.234103 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.232614    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-grpc-tls\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.234103 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.232745    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7f7009b-d19b-43ea-a7d1-1901c90daf79-tmp\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.234103 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.232806    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpggm\" (UniqueName: \"kubernetes.io/projected/d7f7009b-d19b-43ea-a7d1-1901c90daf79-kube-api-access-wpggm\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.234103 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.233942    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7f7009b-d19b-43ea-a7d1-1901c90daf79-metrics-client-ca\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.234654 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.234330    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7f7009b-d19b-43ea-a7d1-1901c90daf79-tmp\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.237282 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.237251    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.237432 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.237413    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-grpc-tls\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.237798 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.237768    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.238263 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.238244    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.238358 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.238293    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.239824 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.239800    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d7f7009b-d19b-43ea-a7d1-1901c90daf79-secret-thanos-querier-tls\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.258076 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.257999    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpggm\" (UniqueName: \"kubernetes.io/projected/d7f7009b-d19b-43ea-a7d1-1901c90daf79-kube-api-access-wpggm\") pod \"thanos-querier-7595945dc8-drcrc\" (UID: \"d7f7009b-d19b-43ea-a7d1-1901c90daf79\") " pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.391512 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.391481    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:01.741511 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:01.741480    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7595945dc8-drcrc"]
Oct 30 02:56:01.745398 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:56:01.745369    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f7009b_d19b_43ea_a7d1_1901c90daf79.slice/crio-a498d69db4350793a64209707e1f0f3d94ec56b32ab55d53f2fa083e39d20a3a WatchSource:0}: Error finding container a498d69db4350793a64209707e1f0f3d94ec56b32ab55d53f2fa083e39d20a3a: Status 404 returned error can't find the container with id a498d69db4350793a64209707e1f0f3d94ec56b32ab55d53f2fa083e39d20a3a
Oct 30 02:56:02.101313 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:02.101273    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc" event={"ID":"d7f7009b-d19b-43ea-a7d1-1901c90daf79","Type":"ContainerStarted","Data":"a498d69db4350793a64209707e1f0f3d94ec56b32ab55d53f2fa083e39d20a3a"}
Oct 30 02:56:02.104628 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:02.103986    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4" event={"ID":"e41def15-5a16-4b6d-b79a-362c08010b6b","Type":"ContainerStarted","Data":"4c0158ead24c88faabfb20f0edd02be39c3d23de4e5a96c61849761241baf362"}
Oct 30 02:56:02.104628 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:02.104059    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4" event={"ID":"e41def15-5a16-4b6d-b79a-362c08010b6b","Type":"ContainerStarted","Data":"9c54389730cf06f095e4074834dabd2f51b47cc60dc1a3b448d914a9a69b4acf"}
Oct 30 02:56:02.104628 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:02.104087    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4" event={"ID":"e41def15-5a16-4b6d-b79a-362c08010b6b","Type":"ContainerStarted","Data":"8ece669e2ad152e91d1cc2936ebbfcc314d5b02fc086303e575c24c6bdc382f9"}
Oct 30 02:56:02.106562 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:02.106521    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r" event={"ID":"bf9aaba6-e684-4534-bbf9-4396fc97797a","Type":"ContainerStarted","Data":"77db6021afa63ea34b5b50cc589add8de98c98e81a0a692133d821b8d31ec6be"}
Oct 30 02:56:02.111244 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:02.111207    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sfkc2" event={"ID":"8386b555-b07c-4481-9226-4a65739fd8db","Type":"ContainerStarted","Data":"5e7b1eead72322e7094389d533e16ba0e1aa91efb4b978a6ae7fea198ed023f9"}
Oct 30 02:56:02.111389 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:02.111260    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sfkc2" event={"ID":"8386b555-b07c-4481-9226-4a65739fd8db","Type":"ContainerStarted","Data":"77508414180e69b3336684d72eec596819bd12ab8b4ba52c6b15c9d2312ae4ea"}
Oct 30 02:56:02.152986 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:02.152920    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z74j4" podStartSLOduration=2.201183276 podStartE2EDuration="4.152876243s" totalImagesPullingTime="1.951692967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:02.152370907 +0000 UTC m=+26.358551127" watchObservedRunningTime="2026-10-30 02:56:02.152876243 +0000 UTC m=+26.359056463"
Oct 30 02:56:02.200921 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:02.200831    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dw86r" podStartSLOduration=2.346101271 podStartE2EDuration="4.200812578s" totalImagesPullingTime="1.854711307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:02.19981366 +0000 UTC m=+26.405993916" watchObservedRunningTime="2026-10-30 02:56:02.200812578 +0000 UTC m=+26.406992799"
Oct 30 02:56:02.246068 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:02.246003    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sfkc2" podStartSLOduration=2.72880328 podStartE2EDuration="4.245984998s" totalImagesPullingTime="1.412593155s" totalInitContainerRuntime="104.588563ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:02.244588869 +0000 UTC m=+26.450769099" watchObservedRunningTime="2026-10-30 02:56:02.245984998 +0000 UTC m=+26.452165216"
Oct 30 02:56:03.116688 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:03.116652    4494 generic.go:356] "Generic (PLEG): container finished" podID="52ce43e0-2ca8-4559-9fcb-3ddad3b9e169" containerID="7a03cafe9f94675f0d0a1ebba5c7d2df2fe45ec6baea0186aea7ee29763f43bf" exitCode=0
Oct 30 02:56:03.117270 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:03.116766    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169","Type":"ContainerDied","Data":"7a03cafe9f94675f0d0a1ebba5c7d2df2fe45ec6baea0186aea7ee29763f43bf"}
Oct 30 02:56:04.123661 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:04.123623    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc" event={"ID":"d7f7009b-d19b-43ea-a7d1-1901c90daf79","Type":"ContainerStarted","Data":"106c96b0efd5a8e0b440857f4a57558a5b57c03fdb0f3c340a4553416654351d"}
Oct 30 02:56:04.124111 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:04.123717    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc" event={"ID":"d7f7009b-d19b-43ea-a7d1-1901c90daf79","Type":"ContainerStarted","Data":"14c5184e0f6b3359dc8749708f72cd9ad457834dc99ca59fed275d8f0a718bf9"}
Oct 30 02:56:05.133566 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:05.133484    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169","Type":"ContainerStarted","Data":"9cd3cb35e6792e09d38548750fb49e8454d52916ec7fe0226cacb7a4300b25a2"}
Oct 30 02:56:05.134026 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:05.133590    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169","Type":"ContainerStarted","Data":"b566748314c20cc46b0195ae3c9bf3d552b581ff51f2ae630dff77dccd372f3b"}
Oct 30 02:56:05.135793 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:05.135767    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc" event={"ID":"d7f7009b-d19b-43ea-a7d1-1901c90daf79","Type":"ContainerStarted","Data":"990f8119d01f3e54d63c69fbb77f014d3d6bcd0836b4db771231bf131f390b63"}
Oct 30 02:56:06.144486 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:06.144443    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169","Type":"ContainerStarted","Data":"b72e4cb99eda0b7e9e1774be45a42a565b07f62720ff86dd5c5156f4ac783d9d"}
Oct 30 02:56:06.144996 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:06.144531    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169","Type":"ContainerStarted","Data":"314159febefa0df4ea77cc7b1773a20f165a14bdd35be9368c1ece3f64b371c1"}
Oct 30 02:56:06.144996 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:06.144573    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169","Type":"ContainerStarted","Data":"302f8e8a01f3773801663e3d3035951403b64fae5ae5d663762d947e58ec5e6b"}
Oct 30 02:56:06.144996 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:06.144609    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"52ce43e0-2ca8-4559-9fcb-3ddad3b9e169","Type":"ContainerStarted","Data":"5afe3baab78b136ceace0850042abd2753a11144167ddbbe0a943d9b1d7b51d0"}
Oct 30 02:56:06.148204 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:06.148170    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc" event={"ID":"d7f7009b-d19b-43ea-a7d1-1901c90daf79","Type":"ContainerStarted","Data":"179c043b8d67aff367a4d73655a86261395aa202d1b99be31bf911aae98faeda"}
Oct 30 02:56:06.148339 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:06.148253    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc" event={"ID":"d7f7009b-d19b-43ea-a7d1-1901c90daf79","Type":"ContainerStarted","Data":"367f6da24ad19ce4a018813223fc9bf512eafef9360c19005e1fb2036d3ba391"}
Oct 30 02:56:06.148339 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:06.148297    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc" event={"ID":"d7f7009b-d19b-43ea-a7d1-1901c90daf79","Type":"ContainerStarted","Data":"7f6b1f01fb5926d2f53f51a189f7104b3659d7532d3d9bcd8df95fc28f81e996"}
Oct 30 02:56:06.148474 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:06.148342    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:06.279227 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:06.279165    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.309688825 podStartE2EDuration="6.279143833s" totalImagesPullingTime="2.954046324s" totalInitContainerRuntime="15.408684ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:06.225880788 +0000 UTC m=+30.432061003" watchObservedRunningTime="2026-10-30 02:56:06.279143833 +0000 UTC m=+30.485324054"
Oct 30 02:56:06.279399 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:06.279363    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc" podStartSLOduration=2.973923509 podStartE2EDuration="6.279355854s" totalImagesPullingTime="3.305432345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:06.27711569 +0000 UTC m=+30.483295932" watchObservedRunningTime="2026-10-30 02:56:06.279355854 +0000 UTC m=+30.485536072"
Oct 30 02:56:07.678728 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.678690    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-866cf6d967-4jkj7"]
Oct 30 02:56:07.718132 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.718098    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76c7446db8-ttqzf"]
Oct 30 02:56:07.725853 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.725830    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.743341 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.743313    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76c7446db8-ttqzf"]
Oct 30 02:56:07.792739 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.792700    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca959a5a-824b-4297-bb7e-0a135042a84b-tmp\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.792739 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.792736    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-service-ca\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.793047 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.792839    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-oauth-config\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.793047 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.792893    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-serving-cert\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.793047 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.792991    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-oauth-serving-cert\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.793047 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.793027    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2k2\" (UniqueName: \"kubernetes.io/projected/ca959a5a-824b-4297-bb7e-0a135042a84b-kube-api-access-2f2k2\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.793249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.793087    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-console-config\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.793249 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.793123    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-trusted-ca-bundle\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.894113 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.894068    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-oauth-config\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.894113 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.894122    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-serving-cert\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.894418 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.894143    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-oauth-serving-cert\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.894418 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.894219    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2k2\" (UniqueName: \"kubernetes.io/projected/ca959a5a-824b-4297-bb7e-0a135042a84b-kube-api-access-2f2k2\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.894418 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.894270    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-console-config\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.894418 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.894307    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-trusted-ca-bundle\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.894418 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.894361    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca959a5a-824b-4297-bb7e-0a135042a84b-tmp\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.894418 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.894390    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-service-ca\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.894938 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.894891    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca959a5a-824b-4297-bb7e-0a135042a84b-tmp\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.895066 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.894963    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-console-config\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.895066 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.895028    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-oauth-serving-cert\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.895204 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.895184    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-service-ca\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.895291 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.895272    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-trusted-ca-bundle\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.897474 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.897452    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-oauth-config\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.897557 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.897479    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-serving-cert\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:07.917601 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:07.917574    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2k2\" (UniqueName: \"kubernetes.io/projected/ca959a5a-824b-4297-bb7e-0a135042a84b-kube-api-access-2f2k2\") pod \"console-76c7446db8-ttqzf\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") " pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:08.043515 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:08.043475    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:08.161323 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:08.161293    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-dvbg6" event={"ID":"c2c612b6-be82-4c17-a26b-06c6768cae0f","Type":"ContainerStarted","Data":"c6fa533d436ca5b8cb0c4ca2a1f1f5982054f70197f9e45e5013769d4ebb6567"}
Oct 30 02:56:08.161500 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:08.161472    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-dvbg6"
Oct 30 02:56:08.162778 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:08.162753    4494 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-dvbg6 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused" start-of-body=
Oct 30 02:56:08.162938 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:08.162803    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-dvbg6" podUID="c2c612b6-be82-4c17-a26b-06c6768cae0f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.11:8080/\": dial tcp 10.134.0.11:8080: connect: connection refused"
Oct 30 02:56:08.355330 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:08.355229    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-dvbg6" podStartSLOduration=1.927153112 podStartE2EDuration="19.355208135s" totalImagesPullingTime="17.428055023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:08.204766161 +0000 UTC m=+32.410946380" watchObservedRunningTime="2026-10-30 02:56:08.355208135 +0000 UTC m=+32.561388351"
Oct 30 02:56:08.356376 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:08.356349    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76c7446db8-ttqzf"]
Oct 30 02:56:08.358545 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:56:08.358513    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca959a5a_824b_4297_bb7e_0a135042a84b.slice/crio-c37eea0652c503bbbb598e8d3e8544e34c37b5be7ffb1ea6c057e995b4a68a2a WatchSource:0}: Error finding container c37eea0652c503bbbb598e8d3e8544e34c37b5be7ffb1ea6c057e995b4a68a2a: Status 404 returned error can't find the container with id c37eea0652c503bbbb598e8d3e8544e34c37b5be7ffb1ea6c057e995b4a68a2a
Oct 30 02:56:08.379927 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:08.379880    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rjwmx"
Oct 30 02:56:09.175416 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:09.175370    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c7446db8-ttqzf" event={"ID":"ca959a5a-824b-4297-bb7e-0a135042a84b","Type":"ContainerStarted","Data":"c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e"}
Oct 30 02:56:09.176037 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:09.175468    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c7446db8-ttqzf" event={"ID":"ca959a5a-824b-4297-bb7e-0a135042a84b","Type":"ContainerStarted","Data":"c37eea0652c503bbbb598e8d3e8544e34c37b5be7ffb1ea6c057e995b4a68a2a"}
Oct 30 02:56:09.188841 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:09.188802    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-dvbg6"
Oct 30 02:56:09.219862 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:09.219795    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76c7446db8-ttqzf" podStartSLOduration=2.219781617 podStartE2EDuration="2.219781617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:09.219146386 +0000 UTC m=+33.425326611" watchObservedRunningTime="2026-10-30 02:56:09.219781617 +0000 UTC m=+33.425961836"
Oct 30 02:56:10.023614 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:10.023572    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:56:12.160615 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:12.160585    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7595945dc8-drcrc"
Oct 30 02:56:18.044526 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:18.044433    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:18.044526 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:18.044499    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:18.049514 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:18.049491    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:18.218754 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:18.218450    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:56:18.292706 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:18.292665    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-567496975-sqqjl"]
Oct 30 02:56:24.081749 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:56:24.081707    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 02:56:32.700081 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:32.700017    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-866cf6d967-4jkj7" podUID="1b582873-8125-4822-aa57-4a387907ac78" containerName="console" containerID="cri-o://ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac" gracePeriod=15
Oct 30 02:56:32.946145 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:32.946118    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:56:33.033608 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.033574    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b582873-8125-4822-aa57-4a387907ac78-kube-api-access-6dnkt\" (UniqueName: \"kubernetes.io/projected/1b582873-8125-4822-aa57-4a387907ac78-kube-api-access-6dnkt\") pod \"1b582873-8125-4822-aa57-4a387907ac78\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") "
Oct 30 02:56:33.033804 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.033618    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-oauth-config\") pod \"1b582873-8125-4822-aa57-4a387907ac78\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") "
Oct 30 02:56:33.033804 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.033654    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b582873-8125-4822-aa57-4a387907ac78-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b582873-8125-4822-aa57-4a387907ac78-tmp\") pod \"1b582873-8125-4822-aa57-4a387907ac78\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") "
Oct 30 02:56:33.033804 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.033673    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-oauth-serving-cert\") pod \"1b582873-8125-4822-aa57-4a387907ac78\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") "
Oct 30 02:56:33.033804 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.033741    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-service-ca\") pod \"1b582873-8125-4822-aa57-4a387907ac78\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") "
Oct 30 02:56:33.033804 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.033778    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-console-config\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-console-config\") pod \"1b582873-8125-4822-aa57-4a387907ac78\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") "
Oct 30 02:56:33.034116 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.033811    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-trusted-ca-bundle\") pod \"1b582873-8125-4822-aa57-4a387907ac78\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") "
Oct 30 02:56:33.034116 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.033837    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-serving-cert\") pod \"1b582873-8125-4822-aa57-4a387907ac78\" (UID: \"1b582873-8125-4822-aa57-4a387907ac78\") "
Oct 30 02:56:33.034116 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.033995    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b582873-8125-4822-aa57-4a387907ac78-tmp" pod "1b582873-8125-4822-aa57-4a387907ac78" (UID: "1b582873-8125-4822-aa57-4a387907ac78"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:56:33.034228 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.034135    4494 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b582873-8125-4822-aa57-4a387907ac78-tmp\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:33.034836 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.034800    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-trusted-ca-bundle" pod "1b582873-8125-4822-aa57-4a387907ac78" (UID: "1b582873-8125-4822-aa57-4a387907ac78"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:33.036576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.035021    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-service-ca" pod "1b582873-8125-4822-aa57-4a387907ac78" (UID: "1b582873-8125-4822-aa57-4a387907ac78"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:33.036576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.035042    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-oauth-serving-cert" pod "1b582873-8125-4822-aa57-4a387907ac78" (UID: "1b582873-8125-4822-aa57-4a387907ac78"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:33.036576 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.035132    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-console-config" pod "1b582873-8125-4822-aa57-4a387907ac78" (UID: "1b582873-8125-4822-aa57-4a387907ac78"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:33.039923 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.037182    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-serving-cert" pod "1b582873-8125-4822-aa57-4a387907ac78" (UID: "1b582873-8125-4822-aa57-4a387907ac78"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:56:33.040068 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.040029    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-oauth-config" pod "1b582873-8125-4822-aa57-4a387907ac78" (UID: "1b582873-8125-4822-aa57-4a387907ac78"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:56:33.040443 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.040416    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b582873-8125-4822-aa57-4a387907ac78-kube-api-access-6dnkt" pod "1b582873-8125-4822-aa57-4a387907ac78" (UID: "1b582873-8125-4822-aa57-4a387907ac78"). InnerVolumeSpecName "kube-api-access-6dnkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:56:33.135389 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.135352    4494 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-service-ca\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:33.135389 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.135386    4494 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-console-config\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:33.135389 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.135396    4494 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-trusted-ca-bundle\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:33.135636 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.135407    4494 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-serving-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:33.135636 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.135416    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dnkt\" (UniqueName: \"kubernetes.io/projected/1b582873-8125-4822-aa57-4a387907ac78-kube-api-access-6dnkt\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:33.135636 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.135425    4494 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b582873-8125-4822-aa57-4a387907ac78-console-oauth-config\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:33.135636 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.135433    4494 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b582873-8125-4822-aa57-4a387907ac78-oauth-serving-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:33.274057 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.274020    4494 generic.go:356] "Generic (PLEG): container finished" podID="1b582873-8125-4822-aa57-4a387907ac78" containerID="ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac" exitCode=0
Oct 30 02:56:33.274241 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.274080    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866cf6d967-4jkj7"
Oct 30 02:56:33.274241 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.274120    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866cf6d967-4jkj7" event={"ID":"1b582873-8125-4822-aa57-4a387907ac78","Type":"ContainerDied","Data":"ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac"}
Oct 30 02:56:33.274241 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.274198    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866cf6d967-4jkj7" event={"ID":"1b582873-8125-4822-aa57-4a387907ac78","Type":"ContainerDied","Data":"5bfc34fc0617ea4fb7174b2586cd6169d69dbccdac80edbcfb3a6230f5b0580b"}
Oct 30 02:56:33.274241 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.274218    4494 scope.go:122] "RemoveContainer" containerID="ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac"
Oct 30 02:56:33.283736 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.283697    4494 scope.go:122] "RemoveContainer" containerID="ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac"
Oct 30 02:56:33.284029 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:56:33.284000    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac\": container with ID starting with ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac not found: ID does not exist" containerID="ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac"
Oct 30 02:56:33.284088 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.284043    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac"} err="failed to get container status \"ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac\": rpc error: code = NotFound desc = could not find container \"ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac\": container with ID starting with ded886e432d3c9e8c5e17edc0676ddb9793ed42a804d494ddb73d980299dbeac not found: ID does not exist"
Oct 30 02:56:33.343465 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.343431    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-866cf6d967-4jkj7"]
Oct 30 02:56:33.356575 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.356544    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-866cf6d967-4jkj7"]
Oct 30 02:56:33.925578 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:33.925545    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b582873-8125-4822-aa57-4a387907ac78" path="/var/lib/kubelet/pods/1b582873-8125-4822-aa57-4a387907ac78/volumes"
Oct 30 02:56:43.313397 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.313333    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-567496975-sqqjl" podUID="61b7918c-ed94-4a74-9f83-e6cb251a5885" containerName="console" containerID="cri-o://e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec" gracePeriod=15
Oct 30 02:56:43.598245 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.598208    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:56:43.628693 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.628657    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-config\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-config\") pod \"61b7918c-ed94-4a74-9f83-e6cb251a5885\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") "
Oct 30 02:56:43.628925 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.628705    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b7918c-ed94-4a74-9f83-e6cb251a5885-tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b7918c-ed94-4a74-9f83-e6cb251a5885-tmp\") pod \"61b7918c-ed94-4a74-9f83-e6cb251a5885\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") "
Oct 30 02:56:43.628925 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.628745    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-oauth-config\") pod \"61b7918c-ed94-4a74-9f83-e6cb251a5885\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") "
Oct 30 02:56:43.628925 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.628767    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b7918c-ed94-4a74-9f83-e6cb251a5885-kube-api-access-rh9j8\" (UniqueName: \"kubernetes.io/projected/61b7918c-ed94-4a74-9f83-e6cb251a5885-kube-api-access-rh9j8\") pod \"61b7918c-ed94-4a74-9f83-e6cb251a5885\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") "
Oct 30 02:56:43.628925 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.628799    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-serving-cert\") pod \"61b7918c-ed94-4a74-9f83-e6cb251a5885\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") "
Oct 30 02:56:43.628925 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.628816    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-oauth-serving-cert\") pod \"61b7918c-ed94-4a74-9f83-e6cb251a5885\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") "
Oct 30 02:56:43.628925 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.628863    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-service-ca\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-service-ca\") pod \"61b7918c-ed94-4a74-9f83-e6cb251a5885\" (UID: \"61b7918c-ed94-4a74-9f83-e6cb251a5885\") "
Oct 30 02:56:43.629198 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.629114    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b7918c-ed94-4a74-9f83-e6cb251a5885-tmp" pod "61b7918c-ed94-4a74-9f83-e6cb251a5885" (UID: "61b7918c-ed94-4a74-9f83-e6cb251a5885"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:56:43.629198 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.629147    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-config" pod "61b7918c-ed94-4a74-9f83-e6cb251a5885" (UID: "61b7918c-ed94-4a74-9f83-e6cb251a5885"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:43.629397 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.629374    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-service-ca" pod "61b7918c-ed94-4a74-9f83-e6cb251a5885" (UID: "61b7918c-ed94-4a74-9f83-e6cb251a5885"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:43.629397 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.629379    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-oauth-serving-cert" pod "61b7918c-ed94-4a74-9f83-e6cb251a5885" (UID: "61b7918c-ed94-4a74-9f83-e6cb251a5885"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:43.631308 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.631280    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b7918c-ed94-4a74-9f83-e6cb251a5885-kube-api-access-rh9j8" pod "61b7918c-ed94-4a74-9f83-e6cb251a5885" (UID: "61b7918c-ed94-4a74-9f83-e6cb251a5885"). InnerVolumeSpecName "kube-api-access-rh9j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:56:43.631308 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.631306    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-oauth-config" pod "61b7918c-ed94-4a74-9f83-e6cb251a5885" (UID: "61b7918c-ed94-4a74-9f83-e6cb251a5885"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:56:43.631448 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.631382    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-serving-cert" pod "61b7918c-ed94-4a74-9f83-e6cb251a5885" (UID: "61b7918c-ed94-4a74-9f83-e6cb251a5885"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:56:43.729782 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.729746    4494 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61b7918c-ed94-4a74-9f83-e6cb251a5885-tmp\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:43.729782 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.729776    4494 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-oauth-config\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:43.729782 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.729789    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh9j8\" (UniqueName: \"kubernetes.io/projected/61b7918c-ed94-4a74-9f83-e6cb251a5885-kube-api-access-rh9j8\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:43.730048 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.729799    4494 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-serving-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:43.730048 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.729807    4494 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-oauth-serving-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:43.730048 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.729816    4494 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-service-ca\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:43.730048 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:43.729824    4494 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b7918c-ed94-4a74-9f83-e6cb251a5885-console-config\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:44.314432 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:44.314395    4494 generic.go:356] "Generic (PLEG): container finished" podID="61b7918c-ed94-4a74-9f83-e6cb251a5885" containerID="e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec" exitCode=0
Oct 30 02:56:44.314853 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:44.314441    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567496975-sqqjl"
Oct 30 02:56:44.314853 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:44.314467    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567496975-sqqjl" event={"ID":"61b7918c-ed94-4a74-9f83-e6cb251a5885","Type":"ContainerDied","Data":"e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec"}
Oct 30 02:56:44.314853 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:44.314518    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567496975-sqqjl" event={"ID":"61b7918c-ed94-4a74-9f83-e6cb251a5885","Type":"ContainerDied","Data":"1e371d05cbc10b25dbdc85c1b4efb9698d53cdc09656fd6c1e216ce1b4a69ff6"}
Oct 30 02:56:44.314853 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:44.314539    4494 scope.go:122] "RemoveContainer" containerID="e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec"
Oct 30 02:56:44.323007 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:44.322990    4494 scope.go:122] "RemoveContainer" containerID="e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec"
Oct 30 02:56:44.323295 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:56:44.323276    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec\": container with ID starting with e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec not found: ID does not exist" containerID="e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec"
Oct 30 02:56:44.323341 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:44.323305    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec"} err="failed to get container status \"e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec\": rpc error: code = NotFound desc = could not find container \"e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec\": container with ID starting with e2e60e0da3e1eb3d5b0327005a861b660933fafeac37ff4884d5fd1f54dab2ec not found: ID does not exist"
Oct 30 02:56:44.365693 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:44.365654    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-567496975-sqqjl"]
Oct 30 02:56:44.378453 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:44.378419    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-567496975-sqqjl"]
Oct 30 02:56:45.935118 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:45.935068    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b7918c-ed94-4a74-9f83-e6cb251a5885" path="/var/lib/kubelet/pods/61b7918c-ed94-4a74-9f83-e6cb251a5885/volumes"
Oct 30 02:56:54.081243 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:56:54.081206    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 02:56:56.689181 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:56.689143    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-clqt4-925xq"]
Oct 30 02:56:56.689717 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:56.689377    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-clqt4-925xq" podUID="e18c7963-1218-4b83-837b-391b7d3edd22" containerName="insights-gathering" containerID="cri-o://88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc" gracePeriod=30
Oct 30 02:56:56.978872 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:56.978840    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:56:57.025660 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.025621    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e18c7963-1218-4b83-837b-391b7d3edd22-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e18c7963-1218-4b83-837b-391b7d3edd22-archives-path\") pod \"e18c7963-1218-4b83-837b-391b7d3edd22\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") "
Oct 30 02:56:57.025660 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.025664    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-trusted-ca-bundle\") pod \"e18c7963-1218-4b83-837b-391b7d3edd22\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") "
Oct 30 02:56:57.025950 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.025751    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-service-ca-bundle\") pod \"e18c7963-1218-4b83-837b-391b7d3edd22\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") "
Oct 30 02:56:57.025950 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.025785    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18c7963-1218-4b83-837b-391b7d3edd22-kube-api-access-p8vxl\" (UniqueName: \"kubernetes.io/projected/e18c7963-1218-4b83-837b-391b7d3edd22-kube-api-access-p8vxl\") pod \"e18c7963-1218-4b83-837b-391b7d3edd22\" (UID: \"e18c7963-1218-4b83-837b-391b7d3edd22\") "
Oct 30 02:56:57.026065 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.025951    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e18c7963-1218-4b83-837b-391b7d3edd22-archives-path" pod "e18c7963-1218-4b83-837b-391b7d3edd22" (UID: "e18c7963-1218-4b83-837b-391b7d3edd22"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:56:57.026120 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.026076    4494 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e18c7963-1218-4b83-837b-391b7d3edd22-archives-path\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.026120 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.026099    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-trusted-ca-bundle" pod "e18c7963-1218-4b83-837b-391b7d3edd22" (UID: "e18c7963-1218-4b83-837b-391b7d3edd22"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:57.026222 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.026141    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-service-ca-bundle" pod "e18c7963-1218-4b83-837b-391b7d3edd22" (UID: "e18c7963-1218-4b83-837b-391b7d3edd22"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:57.028388 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.028365    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18c7963-1218-4b83-837b-391b7d3edd22-kube-api-access-p8vxl" pod "e18c7963-1218-4b83-837b-391b7d3edd22" (UID: "e18c7963-1218-4b83-837b-391b7d3edd22"). InnerVolumeSpecName "kube-api-access-p8vxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:56:57.127279 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.127241    4494 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-service-ca-bundle\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.127279 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.127273    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8vxl\" (UniqueName: \"kubernetes.io/projected/e18c7963-1218-4b83-837b-391b7d3edd22-kube-api-access-p8vxl\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.127279 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.127283    4494 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e18c7963-1218-4b83-837b-391b7d3edd22-trusted-ca-bundle\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.362301 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.362262    4494 generic.go:356] "Generic (PLEG): container finished" podID="e18c7963-1218-4b83-837b-391b7d3edd22" containerID="88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc" exitCode=2
Oct 30 02:56:57.362301 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.362304    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-clqt4-925xq"
Oct 30 02:56:57.362535 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.362342    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-clqt4-925xq" event={"ID":"e18c7963-1218-4b83-837b-391b7d3edd22","Type":"ContainerDied","Data":"88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc"}
Oct 30 02:56:57.362535 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.362392    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-clqt4-925xq" event={"ID":"e18c7963-1218-4b83-837b-391b7d3edd22","Type":"ContainerDied","Data":"15f683ad8289c1a7eab25fbfddc39ff6ec5f73becd79c0e5e6bb7dc6ccc71690"}
Oct 30 02:56:57.362535 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.362415    4494 scope.go:122] "RemoveContainer" containerID="88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc"
Oct 30 02:56:57.371801 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.371784    4494 scope.go:122] "RemoveContainer" containerID="88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc"
Oct 30 02:56:57.372055 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:56:57.372037    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc\": container with ID starting with 88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc not found: ID does not exist" containerID="88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc"
Oct 30 02:56:57.372101 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.372067    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc"} err="failed to get container status \"88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc\": rpc error: code = NotFound desc = could not find container \"88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc\": container with ID starting with 88675c312a6b21c9329d01f47edc4a34adb140c1b01940ad17f788a4e60e88bc not found: ID does not exist"
Oct 30 02:56:57.415613 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.415581    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-clqt4-925xq"]
Oct 30 02:56:57.423992 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.423962    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-clqt4-925xq"]
Oct 30 02:56:57.926229 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:56:57.926200    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18c7963-1218-4b83-837b-391b7d3edd22" path="/var/lib/kubelet/pods/e18c7963-1218-4b83-837b-391b7d3edd22/volumes"
Oct 30 02:57:04.213197 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.213155    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-745478bcb9-llz65"]
Oct 30 02:57:04.213709 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.213517    4494 cpu_manager.go:414] "containerMap: removing container" podUID="61b7918c-ed94-4a74-9f83-e6cb251a5885" containerName="console"
Oct 30 02:57:04.213709 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.213531    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b7918c-ed94-4a74-9f83-e6cb251a5885" containerName="console"
Oct 30 02:57:04.213709 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.213541    4494 cpu_manager.go:414] "containerMap: removing container" podUID="1b582873-8125-4822-aa57-4a387907ac78" containerName="console"
Oct 30 02:57:04.213709 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.213547    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b582873-8125-4822-aa57-4a387907ac78" containerName="console"
Oct 30 02:57:04.213709 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.213566    4494 cpu_manager.go:414] "containerMap: removing container" podUID="e18c7963-1218-4b83-837b-391b7d3edd22" containerName="insights-gathering"
Oct 30 02:57:04.213709 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.213572    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18c7963-1218-4b83-837b-391b7d3edd22" containerName="insights-gathering"
Oct 30 02:57:04.213709 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.213635    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18c7963-1218-4b83-837b-391b7d3edd22" containerName="insights-gathering"
Oct 30 02:57:04.213709 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.213645    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b7918c-ed94-4a74-9f83-e6cb251a5885" containerName="console"
Oct 30 02:57:04.213709 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.213651    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b582873-8125-4822-aa57-4a387907ac78" containerName="console"
Oct 30 02:57:04.218295 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.218271    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.233460 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.233426    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 02:57:04.233460 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.233426    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 02:57:04.233460 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.233426    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 02:57:04.233658 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.233429    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 02:57:04.245852 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.245766    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pzmbm\""
Oct 30 02:57:04.247944 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.247914    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-745478bcb9-llz65"]
Oct 30 02:57:04.278991 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.278953    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-federate-client-tls\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.278991 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.278993    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lghrk\" (UniqueName: \"kubernetes.io/projected/641b6d8c-11ea-4e24-9645-6c64e3b138ed-kube-api-access-lghrk\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.279226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.279074    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/641b6d8c-11ea-4e24-9645-6c64e3b138ed-telemeter-trusted-ca-bundle\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.279226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.279131    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-secret-telemeter-client\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.279226 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.279186    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.279363 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.279236    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/641b6d8c-11ea-4e24-9645-6c64e3b138ed-metrics-client-ca\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.279363 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.279284    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-telemeter-client-tls\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.279363 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.279304    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/641b6d8c-11ea-4e24-9645-6c64e3b138ed-serving-certs-ca-bundle\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.295401 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.295368    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 02:57:04.308429 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.308397    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 02:57:04.379713 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.379682    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.379841 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.379719    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/641b6d8c-11ea-4e24-9645-6c64e3b138ed-metrics-client-ca\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.379928 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.379855    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-telemeter-client-tls\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.379990 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.379924    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/641b6d8c-11ea-4e24-9645-6c64e3b138ed-serving-certs-ca-bundle\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.380044 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.379987    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-federate-client-tls\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.380044 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.380016    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lghrk\" (UniqueName: \"kubernetes.io/projected/641b6d8c-11ea-4e24-9645-6c64e3b138ed-kube-api-access-lghrk\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.380212 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.380049    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/641b6d8c-11ea-4e24-9645-6c64e3b138ed-telemeter-trusted-ca-bundle\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.380212 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.380093    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-secret-telemeter-client\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.380766 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.380687    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/641b6d8c-11ea-4e24-9645-6c64e3b138ed-serving-certs-ca-bundle\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.380766 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.380719    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/641b6d8c-11ea-4e24-9645-6c64e3b138ed-metrics-client-ca\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.381098 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.381079    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/641b6d8c-11ea-4e24-9645-6c64e3b138ed-telemeter-trusted-ca-bundle\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.382462 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.382436    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-telemeter-client-tls\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.382559 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.382499    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-secret-telemeter-client\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.382877 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.382854    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.382970 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.382922    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/641b6d8c-11ea-4e24-9645-6c64e3b138ed-federate-client-tls\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.404079 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.404041    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lghrk\" (UniqueName: \"kubernetes.io/projected/641b6d8c-11ea-4e24-9645-6c64e3b138ed-kube-api-access-lghrk\") pod \"telemeter-client-745478bcb9-llz65\" (UID: \"641b6d8c-11ea-4e24-9645-6c64e3b138ed\") " pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.533793 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.533760    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-745478bcb9-llz65"
Oct 30 02:57:04.854636 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:04.854548    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-745478bcb9-llz65"]
Oct 30 02:57:04.866153 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:57:04.866116    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641b6d8c_11ea_4e24_9645_6c64e3b138ed.slice/crio-e1494f00b136fe0ab21c52f6c0203ca44139a3fc35bb3598c717846d77de1e9e WatchSource:0}: Error finding container e1494f00b136fe0ab21c52f6c0203ca44139a3fc35bb3598c717846d77de1e9e: Status 404 returned error can't find the container with id e1494f00b136fe0ab21c52f6c0203ca44139a3fc35bb3598c717846d77de1e9e
Oct 30 02:57:05.399508 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:05.399465    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-745478bcb9-llz65" event={"ID":"641b6d8c-11ea-4e24-9645-6c64e3b138ed","Type":"ContainerStarted","Data":"e1494f00b136fe0ab21c52f6c0203ca44139a3fc35bb3598c717846d77de1e9e"}
Oct 30 02:57:07.409750 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.409706    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-745478bcb9-llz65" event={"ID":"641b6d8c-11ea-4e24-9645-6c64e3b138ed","Type":"ContainerStarted","Data":"363886554a8d17ef29f1cf399036e1ca8b8c81415341fa0f822b7833cc14cf3e"}
Oct 30 02:57:07.410195 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.409807    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-745478bcb9-llz65" event={"ID":"641b6d8c-11ea-4e24-9645-6c64e3b138ed","Type":"ContainerStarted","Data":"7a900a8168c8c3332cb2c461ff64762ce3c4304001989fa438732dbb41e867d1"}
Oct 30 02:57:07.410195 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.409837    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-745478bcb9-llz65" event={"ID":"641b6d8c-11ea-4e24-9645-6c64e3b138ed","Type":"ContainerStarted","Data":"d581dba6a0683e7e0af782216d1b40296680b2ae308b1c436c181bdb87b8e500"}
Oct 30 02:57:07.482366 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.482314    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-745478bcb9-llz65" podStartSLOduration=1.928393315 podStartE2EDuration="3.482301211s" totalImagesPullingTime="1.553907896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:07.481603647 +0000 UTC m=+91.687783891" watchObservedRunningTime="2026-10-30 02:57:07.482301211 +0000 UTC m=+91.688481426"
Oct 30 02:57:07.625793 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.625758    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7676f4c7d9-9rfs6"]
Oct 30 02:57:07.629043 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.629026    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.650483 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.650446    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7676f4c7d9-9rfs6"]
Oct 30 02:57:07.708146 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.708065    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-oauth-config\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.708146 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.708101    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-serving-cert\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.708146 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.708127    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-oauth-serving-cert\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.708352 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.708264    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-trusted-ca-bundle\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.708352 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.708300    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhd57\" (UniqueName: \"kubernetes.io/projected/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-kube-api-access-nhd57\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.708352 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.708322    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-tmp\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.708445 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.708427    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-service-ca\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.708479 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.708447    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-config\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.808851 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.808806    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-oauth-serving-cert\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809069 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.808864    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-trusted-ca-bundle\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809069 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.808894    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhd57\" (UniqueName: \"kubernetes.io/projected/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-kube-api-access-nhd57\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809069 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.808944    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-tmp\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809069 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.808972    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-service-ca\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809069 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.808989    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-config\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809336 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.809121    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-oauth-config\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809336 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.809169    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-serving-cert\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809435 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.809400    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-tmp\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809750 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.809720    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-oauth-serving-cert\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809814 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.809768    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-service-ca\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809814 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.809782    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-config\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.809890 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.809871    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-trusted-ca-bundle\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.811968 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.811939    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-serving-cert\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.812098 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.812082    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-oauth-config\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.832584 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.832555    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhd57\" (UniqueName: \"kubernetes.io/projected/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-kube-api-access-nhd57\") pod \"console-7676f4c7d9-9rfs6\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") " pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:07.940671 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:07.940641    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:08.239151 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:08.239122    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7676f4c7d9-9rfs6"]
Oct 30 02:57:08.240575 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:57:08.240544    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a22502d_91f4_4ffe_bae5_a7c9bd223f21.slice/crio-2f0a034472932c9fbb2ea547fa6533acfcbf5177b15fda8e70f5997a14e8c3bf WatchSource:0}: Error finding container 2f0a034472932c9fbb2ea547fa6533acfcbf5177b15fda8e70f5997a14e8c3bf: Status 404 returned error can't find the container with id 2f0a034472932c9fbb2ea547fa6533acfcbf5177b15fda8e70f5997a14e8c3bf
Oct 30 02:57:08.415354 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:08.415316    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7676f4c7d9-9rfs6" event={"ID":"2a22502d-91f4-4ffe-bae5-a7c9bd223f21","Type":"ContainerStarted","Data":"ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9"}
Oct 30 02:57:08.415842 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:08.415367    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7676f4c7d9-9rfs6" event={"ID":"2a22502d-91f4-4ffe-bae5-a7c9bd223f21","Type":"ContainerStarted","Data":"2f0a034472932c9fbb2ea547fa6533acfcbf5177b15fda8e70f5997a14e8c3bf"}
Oct 30 02:57:08.459409 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:08.459356    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7676f4c7d9-9rfs6" podStartSLOduration=1.459341145 podStartE2EDuration="1.459341145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:08.458152148 +0000 UTC m=+92.664332366" watchObservedRunningTime="2026-10-30 02:57:08.459341145 +0000 UTC m=+92.665521363"
Oct 30 02:57:17.945988 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:17.943761    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:17.945988 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:17.945085    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:17.950632 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:17.950608    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:18.474770 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.474731    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:18.550717 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.550685    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76c7446db8-ttqzf"]
Oct 30 02:57:18.674913 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.674870    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-78dcd969cf-qqc8s"]
Oct 30 02:57:18.679376 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.679355    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.719037 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.719000    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-oauth-serving-cert\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.719206 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.719062    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-console-oauth-config\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.719206 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.719111    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-console-config\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.719206 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.719182    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-console-serving-cert\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.719352 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.719218    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-tmp\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.719352 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.719249    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-service-ca\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.719456 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.719378    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-trusted-ca-bundle\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.719509 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.719457    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brcnl\" (UniqueName: \"kubernetes.io/projected/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-kube-api-access-brcnl\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.735081 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.735011    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78dcd969cf-qqc8s"]
Oct 30 02:57:18.820056 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.820022    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-trusted-ca-bundle\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.820240 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.820076    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brcnl\" (UniqueName: \"kubernetes.io/projected/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-kube-api-access-brcnl\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.820240 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.820098    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-oauth-serving-cert\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.820240 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.820146    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-console-oauth-config\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.820240 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.820165    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-console-config\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.820240 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.820190    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-console-serving-cert\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.820240 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.820220    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-tmp\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.820541 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.820244    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-service-ca\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.820724 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.820686    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-tmp\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.821100 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.821074    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-service-ca\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.821100 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.821085    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-oauth-serving-cert\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.821216 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.821006    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-console-config\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.821216 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.821074    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-trusted-ca-bundle\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.822844 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.822820    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-console-oauth-config\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.822963 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.822945    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-console-serving-cert\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.843974 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.843947    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brcnl\" (UniqueName: \"kubernetes.io/projected/46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1-kube-api-access-brcnl\") pod \"console-78dcd969cf-qqc8s\" (UID: \"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1\") " pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:18.991372 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:18.991283    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:19.290677 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:19.290644    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78dcd969cf-qqc8s"]
Oct 30 02:57:19.291938 ip-10-0-130-130 kubenswrapper[4494]: W1030 02:57:19.291909    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d6ab97_46c3_4f46_ad7f_5f2bcd914fe1.slice/crio-65d9cb232fb71d7745c5ce10786efd522eec58d8a4e6928c44d810b021eb24c7 WatchSource:0}: Error finding container 65d9cb232fb71d7745c5ce10786efd522eec58d8a4e6928c44d810b021eb24c7: Status 404 returned error can't find the container with id 65d9cb232fb71d7745c5ce10786efd522eec58d8a4e6928c44d810b021eb24c7
Oct 30 02:57:19.474694 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:19.474660    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78dcd969cf-qqc8s" event={"ID":"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1","Type":"ContainerStarted","Data":"08078b8e3f862edd3982e694192ca76018bb562ff9764897bdbd9e69a4ba6558"}
Oct 30 02:57:19.474883 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:19.474714    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78dcd969cf-qqc8s" event={"ID":"46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1","Type":"ContainerStarted","Data":"65d9cb232fb71d7745c5ce10786efd522eec58d8a4e6928c44d810b021eb24c7"}
Oct 30 02:57:19.518651 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:19.518597    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-78dcd969cf-qqc8s" podStartSLOduration=1.518576734 podStartE2EDuration="1.518576734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:19.518051586 +0000 UTC m=+103.724231804" watchObservedRunningTime="2026-10-30 02:57:19.518576734 +0000 UTC m=+103.724756951"
Oct 30 02:57:24.082635 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:57:24.082599    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 02:57:28.992105 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:28.992065    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:28.992585 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:28.992120    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:28.996844 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:28.996824    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:29.517414 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:29.517386    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78dcd969cf-qqc8s"
Oct 30 02:57:29.602555 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:29.602516    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7676f4c7d9-9rfs6"]
Oct 30 02:57:43.573238 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.573174    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-76c7446db8-ttqzf" podUID="ca959a5a-824b-4297-bb7e-0a135042a84b" containerName="console" containerID="cri-o://c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e" gracePeriod=15
Oct 30 02:57:43.823133 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.823108    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:57:43.933764 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.933690    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca959a5a-824b-4297-bb7e-0a135042a84b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca959a5a-824b-4297-bb7e-0a135042a84b-tmp\") pod \"ca959a5a-824b-4297-bb7e-0a135042a84b\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") "
Oct 30 02:57:43.933764 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.933730    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-service-ca\") pod \"ca959a5a-824b-4297-bb7e-0a135042a84b\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") "
Oct 30 02:57:43.933764 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.933756    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-trusted-ca-bundle\") pod \"ca959a5a-824b-4297-bb7e-0a135042a84b\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") "
Oct 30 02:57:43.934045 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.933788    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-console-config\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-console-config\") pod \"ca959a5a-824b-4297-bb7e-0a135042a84b\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") "
Oct 30 02:57:43.934045 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.933890    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-oauth-config\") pod \"ca959a5a-824b-4297-bb7e-0a135042a84b\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") "
Oct 30 02:57:43.934045 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.933944    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-oauth-serving-cert\") pod \"ca959a5a-824b-4297-bb7e-0a135042a84b\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") "
Oct 30 02:57:43.934045 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.934008    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-serving-cert\") pod \"ca959a5a-824b-4297-bb7e-0a135042a84b\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") "
Oct 30 02:57:43.934045 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.934032    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca959a5a-824b-4297-bb7e-0a135042a84b-tmp" pod "ca959a5a-824b-4297-bb7e-0a135042a84b" (UID: "ca959a5a-824b-4297-bb7e-0a135042a84b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:57:43.934279 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.934089    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca959a5a-824b-4297-bb7e-0a135042a84b-kube-api-access-2f2k2\" (UniqueName: \"kubernetes.io/projected/ca959a5a-824b-4297-bb7e-0a135042a84b-kube-api-access-2f2k2\") pod \"ca959a5a-824b-4297-bb7e-0a135042a84b\" (UID: \"ca959a5a-824b-4297-bb7e-0a135042a84b\") "
Oct 30 02:57:43.934279 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.934252    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-service-ca" pod "ca959a5a-824b-4297-bb7e-0a135042a84b" (UID: "ca959a5a-824b-4297-bb7e-0a135042a84b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:43.934387 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.934376    4494 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca959a5a-824b-4297-bb7e-0a135042a84b-tmp\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:43.934387 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.934379    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-oauth-serving-cert" pod "ca959a5a-824b-4297-bb7e-0a135042a84b" (UID: "ca959a5a-824b-4297-bb7e-0a135042a84b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:43.934496 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.934395    4494 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-service-ca\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:43.934496 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.934410    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-trusted-ca-bundle" pod "ca959a5a-824b-4297-bb7e-0a135042a84b" (UID: "ca959a5a-824b-4297-bb7e-0a135042a84b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:43.934605 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.934559    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-console-config" pod "ca959a5a-824b-4297-bb7e-0a135042a84b" (UID: "ca959a5a-824b-4297-bb7e-0a135042a84b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:43.936250 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.936219    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-oauth-config" pod "ca959a5a-824b-4297-bb7e-0a135042a84b" (UID: "ca959a5a-824b-4297-bb7e-0a135042a84b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:43.936375 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.936330    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca959a5a-824b-4297-bb7e-0a135042a84b-kube-api-access-2f2k2" pod "ca959a5a-824b-4297-bb7e-0a135042a84b" (UID: "ca959a5a-824b-4297-bb7e-0a135042a84b"). InnerVolumeSpecName "kube-api-access-2f2k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:43.936436 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:43.936378    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-serving-cert" pod "ca959a5a-824b-4297-bb7e-0a135042a84b" (UID: "ca959a5a-824b-4297-bb7e-0a135042a84b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:44.035018 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.034976    4494 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-serving-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.035018 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.035011    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f2k2\" (UniqueName: \"kubernetes.io/projected/ca959a5a-824b-4297-bb7e-0a135042a84b-kube-api-access-2f2k2\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.035018 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.035034    4494 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-trusted-ca-bundle\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.035274 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.035048    4494 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-console-config\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.035274 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.035061    4494 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ca959a5a-824b-4297-bb7e-0a135042a84b-console-oauth-config\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.035274 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.035072    4494 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ca959a5a-824b-4297-bb7e-0a135042a84b-oauth-serving-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:44.576430 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.576390    4494 generic.go:356] "Generic (PLEG): container finished" podID="ca959a5a-824b-4297-bb7e-0a135042a84b" containerID="c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e" exitCode=0
Oct 30 02:57:44.576884 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.576449    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c7446db8-ttqzf"
Oct 30 02:57:44.576884 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.576476    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c7446db8-ttqzf" event={"ID":"ca959a5a-824b-4297-bb7e-0a135042a84b","Type":"ContainerDied","Data":"c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e"}
Oct 30 02:57:44.576884 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.576536    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c7446db8-ttqzf" event={"ID":"ca959a5a-824b-4297-bb7e-0a135042a84b","Type":"ContainerDied","Data":"c37eea0652c503bbbb598e8d3e8544e34c37b5be7ffb1ea6c057e995b4a68a2a"}
Oct 30 02:57:44.576884 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.576563    4494 scope.go:122] "RemoveContainer" containerID="c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e"
Oct 30 02:57:44.585274 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.585251    4494 scope.go:122] "RemoveContainer" containerID="c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e"
Oct 30 02:57:44.585630 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:57:44.585610    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e\": container with ID starting with c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e not found: ID does not exist" containerID="c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e"
Oct 30 02:57:44.585678 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.585641    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e"} err="failed to get container status \"c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e\": rpc error: code = NotFound desc = could not find container \"c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e\": container with ID starting with c436a0e3c321c3f6c4ec40e028704ab101a5e753f07eb844e65e16d9a07dbc9e not found: ID does not exist"
Oct 30 02:57:44.628745 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.628710    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76c7446db8-ttqzf"]
Oct 30 02:57:44.641175 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:44.641138    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76c7446db8-ttqzf"]
Oct 30 02:57:45.927789 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:45.927745    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca959a5a-824b-4297-bb7e-0a135042a84b" path="/var/lib/kubelet/pods/ca959a5a-824b-4297-bb7e-0a135042a84b/volumes"
Oct 30 02:57:54.081469 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:57:54.081383    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 02:57:54.623918 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:54.623834    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7676f4c7d9-9rfs6" podUID="2a22502d-91f4-4ffe-bae5-a7c9bd223f21" containerName="console" containerID="cri-o://ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9" gracePeriod=15
Oct 30 02:57:54.879645 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:54.879582    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:54.926084 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:54.926050    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-service-ca\") pod \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") "
Oct 30 02:57:54.926251 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:54.926127    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-oauth-config\") pod \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") "
Oct 30 02:57:54.926251 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:54.926151    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-config\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-config\") pod \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") "
Oct 30 02:57:54.926251 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:54.926180    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-trusted-ca-bundle\") pod \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") "
Oct 30 02:57:54.926592 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:54.926494    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-service-ca" pod "2a22502d-91f4-4ffe-bae5-a7c9bd223f21" (UID: "2a22502d-91f4-4ffe-bae5-a7c9bd223f21"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:54.926690 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:54.926599    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-config" pod "2a22502d-91f4-4ffe-bae5-a7c9bd223f21" (UID: "2a22502d-91f4-4ffe-bae5-a7c9bd223f21"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:54.926690 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:54.926660    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-trusted-ca-bundle" pod "2a22502d-91f4-4ffe-bae5-a7c9bd223f21" (UID: "2a22502d-91f4-4ffe-bae5-a7c9bd223f21"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:54.929322 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:54.929296    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-oauth-config" pod "2a22502d-91f4-4ffe-bae5-a7c9bd223f21" (UID: "2a22502d-91f4-4ffe-bae5-a7c9bd223f21"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:55.026657 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.026621    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-serving-cert\") pod \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") "
Oct 30 02:57:55.026657 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.026667    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-kube-api-access-nhd57\" (UniqueName: \"kubernetes.io/projected/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-kube-api-access-nhd57\") pod \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") "
Oct 30 02:57:55.026885 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.026698    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-tmp\") pod \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") "
Oct 30 02:57:55.026885 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.026800    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-oauth-serving-cert\") pod \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\" (UID: \"2a22502d-91f4-4ffe-bae5-a7c9bd223f21\") "
Oct 30 02:57:55.027008 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.026989    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-tmp" pod "2a22502d-91f4-4ffe-bae5-a7c9bd223f21" (UID: "2a22502d-91f4-4ffe-bae5-a7c9bd223f21"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:57:55.027135 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.027116    4494 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-oauth-config\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:55.027135 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.027134    4494 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-config\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:55.027253 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.027145    4494 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-trusted-ca-bundle\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:55.027253 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.027155    4494 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-tmp\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:55.027253 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.027163    4494 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-service-ca\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:55.027253 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.027188    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-oauth-serving-cert" pod "2a22502d-91f4-4ffe-bae5-a7c9bd223f21" (UID: "2a22502d-91f4-4ffe-bae5-a7c9bd223f21"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:57:55.028974 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.028954    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-serving-cert" pod "2a22502d-91f4-4ffe-bae5-a7c9bd223f21" (UID: "2a22502d-91f4-4ffe-bae5-a7c9bd223f21"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:57:55.029081 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.029059    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-kube-api-access-nhd57" pod "2a22502d-91f4-4ffe-bae5-a7c9bd223f21" (UID: "2a22502d-91f4-4ffe-bae5-a7c9bd223f21"). InnerVolumeSpecName "kube-api-access-nhd57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:57:55.128279 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.128242    4494 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-console-serving-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:55.128279 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.128275    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhd57\" (UniqueName: \"kubernetes.io/projected/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-kube-api-access-nhd57\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:55.128279 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.128286    4494 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a22502d-91f4-4ffe-bae5-a7c9bd223f21-oauth-serving-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 02:57:55.618370 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.618329    4494 generic.go:356] "Generic (PLEG): container finished" podID="2a22502d-91f4-4ffe-bae5-a7c9bd223f21" containerID="ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9" exitCode=0
Oct 30 02:57:55.618571 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.618381    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7676f4c7d9-9rfs6"
Oct 30 02:57:55.618571 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.618427    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7676f4c7d9-9rfs6" event={"ID":"2a22502d-91f4-4ffe-bae5-a7c9bd223f21","Type":"ContainerDied","Data":"ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9"}
Oct 30 02:57:55.618571 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.618476    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7676f4c7d9-9rfs6" event={"ID":"2a22502d-91f4-4ffe-bae5-a7c9bd223f21","Type":"ContainerDied","Data":"2f0a034472932c9fbb2ea547fa6533acfcbf5177b15fda8e70f5997a14e8c3bf"}
Oct 30 02:57:55.618571 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.618494    4494 scope.go:122] "RemoveContainer" containerID="ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9"
Oct 30 02:57:55.626739 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.626597    4494 scope.go:122] "RemoveContainer" containerID="ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9"
Oct 30 02:57:55.626868 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:57:55.626850    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9\": container with ID starting with ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9 not found: ID does not exist" containerID="ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9"
Oct 30 02:57:55.627014 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.626878    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9"} err="failed to get container status \"ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9\": rpc error: code = NotFound desc = could not find container \"ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9\": container with ID starting with ef2bd2f1543399b9c7380f4a9aeb22e4db5ac4916ecb30d9877b88abaf13c9b9 not found: ID does not exist"
Oct 30 02:57:55.668971 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.668941    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7676f4c7d9-9rfs6"]
Oct 30 02:57:55.681264 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.681239    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7676f4c7d9-9rfs6"]
Oct 30 02:57:55.927394 ip-10-0-130-130 kubenswrapper[4494]: I1030 02:57:55.927322    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a22502d-91f4-4ffe-bae5-a7c9bd223f21" path="/var/lib/kubelet/pods/2a22502d-91f4-4ffe-bae5-a7c9bd223f21/volumes"
Oct 30 02:58:24.081651 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:58:24.081608    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 02:58:54.081276 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:58:54.081238    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 02:59:24.082035 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:59:24.081957    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 02:59:54.082272 ip-10-0-130-130 kubenswrapper[4494]: E1030 02:59:54.082230    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:00:24.080740 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:00:24.080706    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:00:54.083088 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:00:54.082995    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:01:24.082309 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:01:24.082270    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:01:54.081369 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:01:54.081332    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:02:24.081077 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:02:24.080994    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:02:54.081373 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:02:54.081338    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:03:24.082479 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:03:24.082433    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:03:54.082648 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:03:54.082565    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:04:24.082757 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:04:24.082716    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:04:54.082864 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:04:54.082824    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:05:24.082836 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:05:24.082676    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:05:54.081095 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:05:54.081053    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:06:24.082098 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:06:24.082062    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:06:54.081716 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:06:54.081631    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:07:24.081375 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:07:24.081334    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:07:54.081919 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:07:54.081860    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:08:24.081931 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:08:24.081836    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:08:54.081099 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:08:54.081060    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:09:24.081732 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:09:24.081693    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:09:54.082096 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:09:54.082002    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:10:13.509844 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.509813    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"]
Oct 30 03:10:13.510274 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.510131    4494 cpu_manager.go:414] "containerMap: removing container" podUID="ca959a5a-824b-4297-bb7e-0a135042a84b" containerName="console"
Oct 30 03:10:13.510274 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.510142    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca959a5a-824b-4297-bb7e-0a135042a84b" containerName="console"
Oct 30 03:10:13.510274 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.510161    4494 cpu_manager.go:414] "containerMap: removing container" podUID="2a22502d-91f4-4ffe-bae5-a7c9bd223f21" containerName="console"
Oct 30 03:10:13.510274 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.510167    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a22502d-91f4-4ffe-bae5-a7c9bd223f21" containerName="console"
Oct 30 03:10:13.510274 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.510220    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a22502d-91f4-4ffe-bae5-a7c9bd223f21" containerName="console"
Oct 30 03:10:13.510274 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.510231    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca959a5a-824b-4297-bb7e-0a135042a84b" containerName="console"
Oct 30 03:10:13.513144 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.513128    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.539174 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.539106    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Oct 30 03:10:13.551658 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.551627    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-8g7mn\""
Oct 30 03:10:13.553313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.553277    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"]
Oct 30 03:10:13.555149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.555110    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Oct 30 03:10:13.555300 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.555201    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Oct 30 03:10:13.636246 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.636211    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Oct 30 03:10:13.636789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.636771    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Oct 30 03:10:13.648166 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.648142    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszzs\" (UniqueName: \"kubernetes.io/projected/f0f588a2-775a-493a-8a29-4901be3b1463-kube-api-access-zszzs\") pod \"keda-metrics-apiserver-86bd856cfb-kxhw4\" (UID: \"f0f588a2-775a-493a-8a29-4901be3b1463\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.648242 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.648180    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f0f588a2-775a-493a-8a29-4901be3b1463-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-kxhw4\" (UID: \"f0f588a2-775a-493a-8a29-4901be3b1463\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.648242 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.648201    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f0f588a2-775a-493a-8a29-4901be3b1463-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-kxhw4\" (UID: \"f0f588a2-775a-493a-8a29-4901be3b1463\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.748997 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.748963    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszzs\" (UniqueName: \"kubernetes.io/projected/f0f588a2-775a-493a-8a29-4901be3b1463-kube-api-access-zszzs\") pod \"keda-metrics-apiserver-86bd856cfb-kxhw4\" (UID: \"f0f588a2-775a-493a-8a29-4901be3b1463\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.749219 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.749007    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f0f588a2-775a-493a-8a29-4901be3b1463-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-kxhw4\" (UID: \"f0f588a2-775a-493a-8a29-4901be3b1463\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.749219 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.749040    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f0f588a2-775a-493a-8a29-4901be3b1463-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-kxhw4\" (UID: \"f0f588a2-775a-493a-8a29-4901be3b1463\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.749524 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.749506    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/f0f588a2-775a-493a-8a29-4901be3b1463-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-kxhw4\" (UID: \"f0f588a2-775a-493a-8a29-4901be3b1463\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.751758 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.751738    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f0f588a2-775a-493a-8a29-4901be3b1463-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-kxhw4\" (UID: \"f0f588a2-775a-493a-8a29-4901be3b1463\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.772892 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.772815    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszzs\" (UniqueName: \"kubernetes.io/projected/f0f588a2-775a-493a-8a29-4901be3b1463-kube-api-access-zszzs\") pod \"keda-metrics-apiserver-86bd856cfb-kxhw4\" (UID: \"f0f588a2-775a-493a-8a29-4901be3b1463\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.795152 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.795114    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-cnhtp"]
Oct 30 03:10:13.798423 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.798399    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-cnhtp"
Oct 30 03:10:13.813502 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.813477    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Oct 30 03:10:13.824216 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.824188    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-cnhtp"]
Oct 30 03:10:13.835601 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.835577    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:13.850090 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.850064    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ecb53e1-1abd-402a-aec6-82a7a3b5149a-certificates\") pod \"keda-admission-7b7c985c58-cnhtp\" (UID: \"8ecb53e1-1abd-402a-aec6-82a7a3b5149a\") " pod="openshift-keda/keda-admission-7b7c985c58-cnhtp"
Oct 30 03:10:13.850206 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.850189    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvmb\" (UniqueName: \"kubernetes.io/projected/8ecb53e1-1abd-402a-aec6-82a7a3b5149a-kube-api-access-npvmb\") pod \"keda-admission-7b7c985c58-cnhtp\" (UID: \"8ecb53e1-1abd-402a-aec6-82a7a3b5149a\") " pod="openshift-keda/keda-admission-7b7c985c58-cnhtp"
Oct 30 03:10:13.951283 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.951243    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ecb53e1-1abd-402a-aec6-82a7a3b5149a-certificates\") pod \"keda-admission-7b7c985c58-cnhtp\" (UID: \"8ecb53e1-1abd-402a-aec6-82a7a3b5149a\") " pod="openshift-keda/keda-admission-7b7c985c58-cnhtp"
Oct 30 03:10:13.951466 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.951300    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvmb\" (UniqueName: \"kubernetes.io/projected/8ecb53e1-1abd-402a-aec6-82a7a3b5149a-kube-api-access-npvmb\") pod \"keda-admission-7b7c985c58-cnhtp\" (UID: \"8ecb53e1-1abd-402a-aec6-82a7a3b5149a\") " pod="openshift-keda/keda-admission-7b7c985c58-cnhtp"
Oct 30 03:10:13.953839 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.953816    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ecb53e1-1abd-402a-aec6-82a7a3b5149a-certificates\") pod \"keda-admission-7b7c985c58-cnhtp\" (UID: \"8ecb53e1-1abd-402a-aec6-82a7a3b5149a\") " pod="openshift-keda/keda-admission-7b7c985c58-cnhtp"
Oct 30 03:10:13.976537 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:13.976499    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvmb\" (UniqueName: \"kubernetes.io/projected/8ecb53e1-1abd-402a-aec6-82a7a3b5149a-kube-api-access-npvmb\") pod \"keda-admission-7b7c985c58-cnhtp\" (UID: \"8ecb53e1-1abd-402a-aec6-82a7a3b5149a\") " pod="openshift-keda/keda-admission-7b7c985c58-cnhtp"
Oct 30 03:10:14.110756 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:14.110724    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-cnhtp"
Oct 30 03:10:14.136248 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:14.136223    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"]
Oct 30 03:10:14.137109 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:10:14.137085    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f588a2_775a_493a_8a29_4901be3b1463.slice/crio-3da64353cf79dbf3568b7c1248d1b3866e3b824375a735aa1167b3d9733a9151 WatchSource:0}: Error finding container 3da64353cf79dbf3568b7c1248d1b3866e3b824375a735aa1167b3d9733a9151: Status 404 returned error can't find the container with id 3da64353cf79dbf3568b7c1248d1b3866e3b824375a735aa1167b3d9733a9151
Oct 30 03:10:14.139105 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:14.139086    4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:10:14.292690 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:14.292652    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4" event={"ID":"f0f588a2-775a-493a-8a29-4901be3b1463","Type":"ContainerStarted","Data":"3da64353cf79dbf3568b7c1248d1b3866e3b824375a735aa1167b3d9733a9151"}
Oct 30 03:10:14.405916 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:14.405863    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-cnhtp"]
Oct 30 03:10:14.406979 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:10:14.406952    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecb53e1_1abd_402a_aec6_82a7a3b5149a.slice/crio-9bc8a954b13d9c982ec80485bc9a10921a763fd481b3e32617d0e753b2335372 WatchSource:0}: Error finding container 9bc8a954b13d9c982ec80485bc9a10921a763fd481b3e32617d0e753b2335372: Status 404 returned error can't find the container with id 9bc8a954b13d9c982ec80485bc9a10921a763fd481b3e32617d0e753b2335372
Oct 30 03:10:15.300881 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:15.300841    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-cnhtp" event={"ID":"8ecb53e1-1abd-402a-aec6-82a7a3b5149a","Type":"ContainerStarted","Data":"9bc8a954b13d9c982ec80485bc9a10921a763fd481b3e32617d0e753b2335372"}
Oct 30 03:10:18.316772 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:18.316720    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4" event={"ID":"f0f588a2-775a-493a-8a29-4901be3b1463","Type":"ContainerStarted","Data":"0c54948efeeb4fe1fc31ae3cd9d1db335d923fbbb6d234b3ef85f1f3f02f8c8e"}
Oct 30 03:10:18.317237 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:18.316878    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:18.318620 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:18.318590    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-cnhtp" event={"ID":"8ecb53e1-1abd-402a-aec6-82a7a3b5149a","Type":"ContainerStarted","Data":"9d170fd2d952c4f6f04683bf45a059fa90b03998517f61741f2ceb96bcb54c13"}
Oct 30 03:10:18.318741 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:18.318673    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-7b7c985c58-cnhtp"
Oct 30 03:10:18.359252 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:18.359203    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4" podStartSLOduration=1.800369668 podStartE2EDuration="5.359186562s" totalImagesPullingTime="3.558816894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:18.358403645 +0000 UTC m=+882.564583863" watchObservedRunningTime="2026-10-30 03:10:18.359186562 +0000 UTC m=+882.565366782"
Oct 30 03:10:18.399274 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:18.399223    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-admission-7b7c985c58-cnhtp" podStartSLOduration=2.32346382 podStartE2EDuration="5.399208696s" totalImagesPullingTime="3.075744876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:18.39864402 +0000 UTC m=+882.604824275" watchObservedRunningTime="2026-10-30 03:10:18.399208696 +0000 UTC m=+882.605388933"
Oct 30 03:10:24.082256 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:10:24.082214    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:10:29.326929 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:29.326884    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-kxhw4"
Oct 30 03:10:39.325053 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:10:39.325013    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-7b7c985c58-cnhtp"
Oct 30 03:10:54.080869 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:10:54.080830    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:11:24.082404 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:11:24.082364    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:11:29.046190 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.046153    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"]
Oct 30 03:11:29.048311 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.048293    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"
Oct 30 03:11:29.063434 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.063405    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 03:11:29.063434 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.063422    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 03:11:29.075656 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.075629    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gmhzg\""
Oct 30 03:11:29.077084 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.077060    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"]
Oct 30 03:11:29.197953 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.197921    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5qck\" (UniqueName: \"kubernetes.io/projected/f85ff849-ee76-48cb-aaed-bbe5b7cfbe05-kube-api-access-x5qck\") pod \"cert-manager-cainjector-5c57bbd4fc-8rt84\" (UID: \"f85ff849-ee76-48cb-aaed-bbe5b7cfbe05\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"
Oct 30 03:11:29.198104 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.197961    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f85ff849-ee76-48cb-aaed-bbe5b7cfbe05-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8rt84\" (UID: \"f85ff849-ee76-48cb-aaed-bbe5b7cfbe05\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"
Oct 30 03:11:29.299334 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.299259    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5qck\" (UniqueName: \"kubernetes.io/projected/f85ff849-ee76-48cb-aaed-bbe5b7cfbe05-kube-api-access-x5qck\") pod \"cert-manager-cainjector-5c57bbd4fc-8rt84\" (UID: \"f85ff849-ee76-48cb-aaed-bbe5b7cfbe05\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"
Oct 30 03:11:29.299334 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.299300    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f85ff849-ee76-48cb-aaed-bbe5b7cfbe05-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8rt84\" (UID: \"f85ff849-ee76-48cb-aaed-bbe5b7cfbe05\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"
Oct 30 03:11:29.322750 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.322721    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f85ff849-ee76-48cb-aaed-bbe5b7cfbe05-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8rt84\" (UID: \"f85ff849-ee76-48cb-aaed-bbe5b7cfbe05\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"
Oct 30 03:11:29.322922 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.322791    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5qck\" (UniqueName: \"kubernetes.io/projected/f85ff849-ee76-48cb-aaed-bbe5b7cfbe05-kube-api-access-x5qck\") pod \"cert-manager-cainjector-5c57bbd4fc-8rt84\" (UID: \"f85ff849-ee76-48cb-aaed-bbe5b7cfbe05\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"
Oct 30 03:11:29.370602 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.370577    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"
Oct 30 03:11:29.661174 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:29.661145    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84"]
Oct 30 03:11:29.662278 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:11:29.662247    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85ff849_ee76_48cb_aaed_bbe5b7cfbe05.slice/crio-d267ec1f79f566e59934e70ec590d0964a68dbc2ab076328e46adb68467ff5f3 WatchSource:0}: Error finding container d267ec1f79f566e59934e70ec590d0964a68dbc2ab076328e46adb68467ff5f3: Status 404 returned error can't find the container with id d267ec1f79f566e59934e70ec590d0964a68dbc2ab076328e46adb68467ff5f3
Oct 30 03:11:30.602210 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:30.602170    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84" event={"ID":"f85ff849-ee76-48cb-aaed-bbe5b7cfbe05","Type":"ContainerStarted","Data":"d267ec1f79f566e59934e70ec590d0964a68dbc2ab076328e46adb68467ff5f3"}
Oct 30 03:11:33.617196 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:33.617160    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84" event={"ID":"f85ff849-ee76-48cb-aaed-bbe5b7cfbe05","Type":"ContainerStarted","Data":"ebbfea5df569f2dd69a9eaef7dd832926e07c55ae6c1f7ec2062197eea79e3b2"}
Oct 30 03:11:33.657811 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:33.657759    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8rt84" podStartSLOduration=1.6275984810000002 podStartE2EDuration="4.657744919s" totalImagesPullingTime="3.030146438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:33.657299412 +0000 UTC m=+957.863479632" watchObservedRunningTime="2026-10-30 03:11:33.657744919 +0000 UTC m=+957.863925136"
Oct 30 03:11:46.918491 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:46.918453    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"]
Oct 30 03:11:46.922182 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:46.922155    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"
Oct 30 03:11:46.933981 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:46.933955    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa67590-f8c0-42c1-bd6f-8dfebe36f55c-tmp\") pod \"openshift-lws-operator-8465994787-jt2ng\" (UID: \"8fa67590-f8c0-42c1-bd6f-8dfebe36f55c\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"
Oct 30 03:11:46.934102 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:46.934000    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b674n\" (UniqueName: \"kubernetes.io/projected/8fa67590-f8c0-42c1-bd6f-8dfebe36f55c-kube-api-access-b674n\") pod \"openshift-lws-operator-8465994787-jt2ng\" (UID: \"8fa67590-f8c0-42c1-bd6f-8dfebe36f55c\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"
Oct 30 03:11:46.949762 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:46.949734    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 03:11:46.949953 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:46.949783    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-k74ch\""
Oct 30 03:11:46.949953 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:46.949795    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 03:11:46.951237 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:46.951216    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"]
Oct 30 03:11:47.034805 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.034767    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b674n\" (UniqueName: \"kubernetes.io/projected/8fa67590-f8c0-42c1-bd6f-8dfebe36f55c-kube-api-access-b674n\") pod \"openshift-lws-operator-8465994787-jt2ng\" (UID: \"8fa67590-f8c0-42c1-bd6f-8dfebe36f55c\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"
Oct 30 03:11:47.035036 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.034924    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa67590-f8c0-42c1-bd6f-8dfebe36f55c-tmp\") pod \"openshift-lws-operator-8465994787-jt2ng\" (UID: \"8fa67590-f8c0-42c1-bd6f-8dfebe36f55c\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"
Oct 30 03:11:47.035272 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.035254    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fa67590-f8c0-42c1-bd6f-8dfebe36f55c-tmp\") pod \"openshift-lws-operator-8465994787-jt2ng\" (UID: \"8fa67590-f8c0-42c1-bd6f-8dfebe36f55c\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"
Oct 30 03:11:47.061815 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.061779    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b674n\" (UniqueName: \"kubernetes.io/projected/8fa67590-f8c0-42c1-bd6f-8dfebe36f55c-kube-api-access-b674n\") pod \"openshift-lws-operator-8465994787-jt2ng\" (UID: \"8fa67590-f8c0-42c1-bd6f-8dfebe36f55c\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"
Oct 30 03:11:47.233585 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.233497    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"
Oct 30 03:11:47.294914 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.294855    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-864qn"]
Oct 30 03:11:47.300395 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.300371    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-864qn"
Oct 30 03:11:47.318125 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.318096    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5hqnw\""
Oct 30 03:11:47.318289 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.318258    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-864qn"]
Oct 30 03:11:47.337522 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.337483    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8845620-0f85-49b4-a8e5-494d8dd7d906-bound-sa-token\") pod \"cert-manager-597b95cb84-864qn\" (UID: \"c8845620-0f85-49b4-a8e5-494d8dd7d906\") " pod="cert-manager/cert-manager-597b95cb84-864qn"
Oct 30 03:11:47.337522 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.337521    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv9lm\" (UniqueName: \"kubernetes.io/projected/c8845620-0f85-49b4-a8e5-494d8dd7d906-kube-api-access-cv9lm\") pod \"cert-manager-597b95cb84-864qn\" (UID: \"c8845620-0f85-49b4-a8e5-494d8dd7d906\") " pod="cert-manager/cert-manager-597b95cb84-864qn"
Oct 30 03:11:47.439076 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.439025    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8845620-0f85-49b4-a8e5-494d8dd7d906-bound-sa-token\") pod \"cert-manager-597b95cb84-864qn\" (UID: \"c8845620-0f85-49b4-a8e5-494d8dd7d906\") " pod="cert-manager/cert-manager-597b95cb84-864qn"
Oct 30 03:11:47.439076 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.439082    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9lm\" (UniqueName: \"kubernetes.io/projected/c8845620-0f85-49b4-a8e5-494d8dd7d906-kube-api-access-cv9lm\") pod \"cert-manager-597b95cb84-864qn\" (UID: \"c8845620-0f85-49b4-a8e5-494d8dd7d906\") " pod="cert-manager/cert-manager-597b95cb84-864qn"
Oct 30 03:11:47.464680 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.464650    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c8845620-0f85-49b4-a8e5-494d8dd7d906-bound-sa-token\") pod \"cert-manager-597b95cb84-864qn\" (UID: \"c8845620-0f85-49b4-a8e5-494d8dd7d906\") " pod="cert-manager/cert-manager-597b95cb84-864qn"
Oct 30 03:11:47.465015 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.464995    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9lm\" (UniqueName: \"kubernetes.io/projected/c8845620-0f85-49b4-a8e5-494d8dd7d906-kube-api-access-cv9lm\") pod \"cert-manager-597b95cb84-864qn\" (UID: \"c8845620-0f85-49b4-a8e5-494d8dd7d906\") " pod="cert-manager/cert-manager-597b95cb84-864qn"
Oct 30 03:11:47.528194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.528162    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng"]
Oct 30 03:11:47.529392 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:11:47.529361    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fa67590_f8c0_42c1_bd6f_8dfebe36f55c.slice/crio-d26c1ea6bad0bae2768a1f4f46b5877fa42ddca9da198a03b1383ffa89b6a5d0 WatchSource:0}: Error finding container d26c1ea6bad0bae2768a1f4f46b5877fa42ddca9da198a03b1383ffa89b6a5d0: Status 404 returned error can't find the container with id d26c1ea6bad0bae2768a1f4f46b5877fa42ddca9da198a03b1383ffa89b6a5d0
Oct 30 03:11:47.615172 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.615126    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-864qn"
Oct 30 03:11:47.679636 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.679595    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng" event={"ID":"8fa67590-f8c0-42c1-bd6f-8dfebe36f55c","Type":"ContainerStarted","Data":"d26c1ea6bad0bae2768a1f4f46b5877fa42ddca9da198a03b1383ffa89b6a5d0"}
Oct 30 03:11:47.909927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:47.909882    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-864qn"]
Oct 30 03:11:47.910584 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:11:47.910561    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8845620_0f85_49b4_a8e5_494d8dd7d906.slice/crio-27ab4afaf3719c3e3ebff87204bc5a059cc952362c04797b4799de494405596c WatchSource:0}: Error finding container 27ab4afaf3719c3e3ebff87204bc5a059cc952362c04797b4799de494405596c: Status 404 returned error can't find the container with id 27ab4afaf3719c3e3ebff87204bc5a059cc952362c04797b4799de494405596c
Oct 30 03:11:48.689963 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:48.689926    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-864qn" event={"ID":"c8845620-0f85-49b4-a8e5-494d8dd7d906","Type":"ContainerStarted","Data":"9022f9a742c0a8ed7e85610f851345b3194e84bfcde8e2c3ffa361036eec89f9"}
Oct 30 03:11:48.690401 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:48.689979    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-864qn" event={"ID":"c8845620-0f85-49b4-a8e5-494d8dd7d906","Type":"ContainerStarted","Data":"27ab4afaf3719c3e3ebff87204bc5a059cc952362c04797b4799de494405596c"}
Oct 30 03:11:48.731231 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:48.731167    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-864qn" podStartSLOduration=1.731145876 podStartE2EDuration="1.731145876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:48.730582444 +0000 UTC m=+972.936762663" watchObservedRunningTime="2026-10-30 03:11:48.731145876 +0000 UTC m=+972.937326096"
Oct 30 03:11:50.701196 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:50.701158    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng" event={"ID":"8fa67590-f8c0-42c1-bd6f-8dfebe36f55c","Type":"ContainerStarted","Data":"4e0ea5760e883ab60b542de125d5a69430e681d3849396dca9cb5e56fda8b135"}
Oct 30 03:11:50.742493 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:11:50.742439    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-jt2ng" podStartSLOduration=2.299874882 podStartE2EDuration="4.742425425s" totalImagesPullingTime="2.442550543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:50.742025711 +0000 UTC m=+974.948205929" watchObservedRunningTime="2026-10-30 03:11:50.742425425 +0000 UTC m=+974.948605643"
Oct 30 03:11:54.082136 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:11:54.082099    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:12:12.470985 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.470948    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"]
Oct 30 03:12:12.477165 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.477131    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.492380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.492352    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-pzw79\""
Oct 30 03:12:12.496299 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.496270    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 03:12:12.496437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.496342    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 03:12:12.496437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.496351    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 03:12:12.506683 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.506654    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"]
Oct 30 03:12:12.560835 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.560798    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.561034 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.560841    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.561087 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.561037    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwbq\" (UniqueName: \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-kube-api-access-shwbq\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.561087 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.561079    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.561176 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.561114    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.561221 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.561195    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.561264 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.561249    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.561311 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.561295    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.561350 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.561338    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662060    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwbq\" (UniqueName: \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-kube-api-access-shwbq\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662112    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662388 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662147    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662388 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662169    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662388 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662210    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662388 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662237    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662388 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662261    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662388 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662295    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662388 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662322    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662753 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662725    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662830 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662804    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662886 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662840    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.662996 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.662977    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.663294 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.663269    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.664731 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.664709    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.664828 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.664811    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.685800 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.685773    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.686076 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.686053    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwbq\" (UniqueName: \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-kube-api-access-shwbq\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:12.792151 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:12.792118    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:13.105809 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:13.105779    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"]
Oct 30 03:12:13.107719 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:12:13.107688    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a3121c_9c1e_4085_a4b6_151ccd0438e5.slice/crio-225c885bbaf76cb38d40a32c7dab19759b7a8ee2c38c56ca3e6433d7f51f859f WatchSource:0}: Error finding container 225c885bbaf76cb38d40a32c7dab19759b7a8ee2c38c56ca3e6433d7f51f859f: Status 404 returned error can't find the container with id 225c885bbaf76cb38d40a32c7dab19759b7a8ee2c38c56ca3e6433d7f51f859f
Oct 30 03:12:13.792211 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:13.792168    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" event={"ID":"92a3121c-9c1e-4085-a4b6-151ccd0438e5","Type":"ContainerStarted","Data":"225c885bbaf76cb38d40a32c7dab19759b7a8ee2c38c56ca3e6433d7f51f859f"}
Oct 30 03:12:15.629416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:15.629386    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:12:15.629696 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:15.629455    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:12:15.629696 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:15.629485    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:12:15.803044 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:15.802997    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" event={"ID":"92a3121c-9c1e-4085-a4b6-151ccd0438e5","Type":"ContainerStarted","Data":"951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f"}
Oct 30 03:12:15.877797 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:15.877687    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" podStartSLOduration=1.359182226 podStartE2EDuration="3.877670847s" totalImagesPullingTime="2.518488621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:15.864274914 +0000 UTC m=+1000.070455132" watchObservedRunningTime="2026-10-30 03:12:15.877670847 +0000 UTC m=+1000.083851064"
Oct 30 03:12:16.792612 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:16.792571    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:16.797399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:16.797372    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:16.807446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:16.807424    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:16.808236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:16.808217    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:12:17.595214 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.595177    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"]
Oct 30 03:12:17.597425 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.597407    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.619458 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.619429    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"]
Oct 30 03:12:17.703948 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.703880    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 03:12:17.704154 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.703979    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mbvvd\""
Oct 30 03:12:17.704154 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.703984    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 03:12:17.704154 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.704055    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 03:12:17.707396 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.707368    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/14bfd77d-d373-4545-9d7c-78ea3ebda44b-metrics-cert\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.707523 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.707407    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vbwb\" (UniqueName: \"kubernetes.io/projected/14bfd77d-d373-4545-9d7c-78ea3ebda44b-kube-api-access-9vbwb\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.707523 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.707430    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/14bfd77d-d373-4545-9d7c-78ea3ebda44b-manager-config\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.707523 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.707485    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14bfd77d-d373-4545-9d7c-78ea3ebda44b-cert\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.808524 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.808489    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbwb\" (UniqueName: \"kubernetes.io/projected/14bfd77d-d373-4545-9d7c-78ea3ebda44b-kube-api-access-9vbwb\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.809080 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.808536    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/14bfd77d-d373-4545-9d7c-78ea3ebda44b-manager-config\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.809080 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.808640    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14bfd77d-d373-4545-9d7c-78ea3ebda44b-cert\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.809080 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.808776    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/14bfd77d-d373-4545-9d7c-78ea3ebda44b-metrics-cert\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.809373 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.809344    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/14bfd77d-d373-4545-9d7c-78ea3ebda44b-manager-config\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.811716 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.811661    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/14bfd77d-d373-4545-9d7c-78ea3ebda44b-metrics-cert\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.811951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.811931    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14bfd77d-d373-4545-9d7c-78ea3ebda44b-cert\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.832576 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.832544    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbwb\" (UniqueName: \"kubernetes.io/projected/14bfd77d-d373-4545-9d7c-78ea3ebda44b-kube-api-access-9vbwb\") pod \"lws-controller-manager-6784dd78d9-fzcwb\" (UID: \"14bfd77d-d373-4545-9d7c-78ea3ebda44b\") " pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:17.908682 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:17.908580    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:18.203409 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:18.203384    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"]
Oct 30 03:12:18.203889 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:12:18.203863    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14bfd77d_d373_4545_9d7c_78ea3ebda44b.slice/crio-446ae88352817d774ba127ae09ac52ad64240b7565efaf3f0b80c9e1b46b34a1 WatchSource:0}: Error finding container 446ae88352817d774ba127ae09ac52ad64240b7565efaf3f0b80c9e1b46b34a1: Status 404 returned error can't find the container with id 446ae88352817d774ba127ae09ac52ad64240b7565efaf3f0b80c9e1b46b34a1
Oct 30 03:12:18.819734 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:18.819684    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb" event={"ID":"14bfd77d-d373-4545-9d7c-78ea3ebda44b","Type":"ContainerStarted","Data":"446ae88352817d774ba127ae09ac52ad64240b7565efaf3f0b80c9e1b46b34a1"}
Oct 30 03:12:19.825172 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:19.825070    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb" event={"ID":"14bfd77d-d373-4545-9d7c-78ea3ebda44b","Type":"ContainerStarted","Data":"5aaebeb342d0998c56560eb883babcaa1810c8e629067a1e36d1639a20b2574c"}
Oct 30 03:12:19.825613 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:19.825248    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:19.872459 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:19.872401    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb" podStartSLOduration=1.534623141 podStartE2EDuration="2.872383034s" totalImagesPullingTime="1.337759893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:19.8720793 +0000 UTC m=+1004.078259520" watchObservedRunningTime="2026-10-30 03:12:19.872383034 +0000 UTC m=+1004.078563249"
Oct 30 03:12:24.081861 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:12:24.081821    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:12:30.830831 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:12:30.830801    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6784dd78d9-fzcwb"
Oct 30 03:12:54.081549 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:12:54.081470    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:13:24.081100 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:13:24.081065    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:13:28.556604 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:28.556572    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tjm2"]
Oct 30 03:13:28.559039 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:28.559016    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2tjm2"
Oct 30 03:13:28.585929 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:28.585882    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 03:13:28.586036 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:28.585885    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 03:13:28.586036 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:28.585951    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xksk4\""
Oct 30 03:13:28.587219 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:28.587200    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tjm2"]
Oct 30 03:13:28.607009 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:28.606987    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dll\" (UniqueName: \"kubernetes.io/projected/bf10773f-e4af-49d8-9f53-824cdaea6e1b-kube-api-access-k4dll\") pod \"authorino-57f7f8659f-2tjm2\" (UID: \"bf10773f-e4af-49d8-9f53-824cdaea6e1b\") " pod="kuadrant-system/authorino-57f7f8659f-2tjm2"
Oct 30 03:13:28.707846 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:28.707819    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dll\" (UniqueName: \"kubernetes.io/projected/bf10773f-e4af-49d8-9f53-824cdaea6e1b-kube-api-access-k4dll\") pod \"authorino-57f7f8659f-2tjm2\" (UID: \"bf10773f-e4af-49d8-9f53-824cdaea6e1b\") " pod="kuadrant-system/authorino-57f7f8659f-2tjm2"
Oct 30 03:13:28.730931 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:28.730884    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dll\" (UniqueName: \"kubernetes.io/projected/bf10773f-e4af-49d8-9f53-824cdaea6e1b-kube-api-access-k4dll\") pod \"authorino-57f7f8659f-2tjm2\" (UID: \"bf10773f-e4af-49d8-9f53-824cdaea6e1b\") " pod="kuadrant-system/authorino-57f7f8659f-2tjm2"
Oct 30 03:13:28.869767 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:28.869691    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2tjm2"
Oct 30 03:13:29.161299 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:29.161270    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tjm2"]
Oct 30 03:13:29.161874 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:13:29.161849    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf10773f_e4af_49d8_9f53_824cdaea6e1b.slice/crio-2234285daf4e8b4bc70cbe9d9215d77e7a92dd6cf3c4e4a3a9a03e7418af4c92 WatchSource:0}: Error finding container 2234285daf4e8b4bc70cbe9d9215d77e7a92dd6cf3c4e4a3a9a03e7418af4c92: Status 404 returned error can't find the container with id 2234285daf4e8b4bc70cbe9d9215d77e7a92dd6cf3c4e4a3a9a03e7418af4c92
Oct 30 03:13:30.120574 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:30.120538    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2tjm2" event={"ID":"bf10773f-e4af-49d8-9f53-824cdaea6e1b","Type":"ContainerStarted","Data":"2234285daf4e8b4bc70cbe9d9215d77e7a92dd6cf3c4e4a3a9a03e7418af4c92"}
Oct 30 03:13:32.131542 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:32.131505    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2tjm2" event={"ID":"bf10773f-e4af-49d8-9f53-824cdaea6e1b","Type":"ContainerStarted","Data":"fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371"}
Oct 30 03:13:32.171686 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:32.171589    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-2tjm2" podStartSLOduration=1.43434426 podStartE2EDuration="4.171576474s" totalImagesPullingTime="2.737232214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:32.171076858 +0000 UTC m=+1076.377257077" watchObservedRunningTime="2026-10-30 03:13:32.171576474 +0000 UTC m=+1076.377756691"
Oct 30 03:13:32.176323 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:32.176295    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tjm2"]
Oct 30 03:13:34.140442 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:34.140387    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-2tjm2" podUID="bf10773f-e4af-49d8-9f53-824cdaea6e1b" containerName="authorino" containerID="cri-o://fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371" gracePeriod=30
Oct 30 03:13:34.374370 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:34.374347    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2tjm2"
Oct 30 03:13:34.457504 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:34.457431    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf10773f-e4af-49d8-9f53-824cdaea6e1b-kube-api-access-k4dll\" (UniqueName: \"kubernetes.io/projected/bf10773f-e4af-49d8-9f53-824cdaea6e1b-kube-api-access-k4dll\") pod \"bf10773f-e4af-49d8-9f53-824cdaea6e1b\" (UID: \"bf10773f-e4af-49d8-9f53-824cdaea6e1b\") "
Oct 30 03:13:34.459680 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:34.459659    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf10773f-e4af-49d8-9f53-824cdaea6e1b-kube-api-access-k4dll" pod "bf10773f-e4af-49d8-9f53-824cdaea6e1b" (UID: "bf10773f-e4af-49d8-9f53-824cdaea6e1b"). InnerVolumeSpecName "kube-api-access-k4dll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:13:34.558978 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:34.558941    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4dll\" (UniqueName: \"kubernetes.io/projected/bf10773f-e4af-49d8-9f53-824cdaea6e1b-kube-api-access-k4dll\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:13:35.145881 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:35.145848    4494 generic.go:356] "Generic (PLEG): container finished" podID="bf10773f-e4af-49d8-9f53-824cdaea6e1b" containerID="fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371" exitCode=0
Oct 30 03:13:35.146358 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:35.145892    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-2tjm2"
Oct 30 03:13:35.146358 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:35.145912    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2tjm2" event={"ID":"bf10773f-e4af-49d8-9f53-824cdaea6e1b","Type":"ContainerDied","Data":"fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371"}
Oct 30 03:13:35.146358 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:35.145946    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-2tjm2" event={"ID":"bf10773f-e4af-49d8-9f53-824cdaea6e1b","Type":"ContainerDied","Data":"2234285daf4e8b4bc70cbe9d9215d77e7a92dd6cf3c4e4a3a9a03e7418af4c92"}
Oct 30 03:13:35.146358 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:35.145963    4494 scope.go:122] "RemoveContainer" containerID="fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371"
Oct 30 03:13:35.154749 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:35.154733    4494 scope.go:122] "RemoveContainer" containerID="fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371"
Oct 30 03:13:35.155045 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:13:35.155027    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371\": container with ID starting with fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371 not found: ID does not exist" containerID="fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371"
Oct 30 03:13:35.155093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:35.155053    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371"} err="failed to get container status \"fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371\": rpc error: code = NotFound desc = could not find container \"fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371\": container with ID starting with fb4569209845f63de2c393b7cbe82314ca01587b96f1b80781e9874180248371 not found: ID does not exist"
Oct 30 03:13:35.195877 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:35.195850    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tjm2"]
Oct 30 03:13:35.200243 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:35.200220    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-2tjm2"]
Oct 30 03:13:35.927397 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:35.927367    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf10773f-e4af-49d8-9f53-824cdaea6e1b" path="/var/lib/kubelet/pods/bf10773f-e4af-49d8-9f53-824cdaea6e1b/volumes"
Oct 30 03:13:53.303532 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.303499    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557c8f8584-6lfw5"]
Oct 30 03:13:53.303945 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.303850    4494 cpu_manager.go:414] "containerMap: removing container" podUID="bf10773f-e4af-49d8-9f53-824cdaea6e1b" containerName="authorino"
Oct 30 03:13:53.303945 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.303862    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf10773f-e4af-49d8-9f53-824cdaea6e1b" containerName="authorino"
Oct 30 03:13:53.303945 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.303940    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf10773f-e4af-49d8-9f53-824cdaea6e1b" containerName="authorino"
Oct 30 03:13:53.305791 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.305774    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557c8f8584-6lfw5"
Oct 30 03:13:53.332695 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.332672    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 03:13:53.332804 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.332729    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xksk4\""
Oct 30 03:13:53.332804 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.332732    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 03:13:53.332889 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.332832    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 03:13:53.334018 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.334001    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557c8f8584-6lfw5"]
Oct 30 03:13:53.403839 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.403815    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06f8e1b8-8ddd-4298-9c61-f326e03387a7-tls-cert\") pod \"authorino-557c8f8584-6lfw5\" (UID: \"06f8e1b8-8ddd-4298-9c61-f326e03387a7\") " pod="kuadrant-system/authorino-557c8f8584-6lfw5"
Oct 30 03:13:53.403966 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.403845    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5prb\" (UniqueName: \"kubernetes.io/projected/06f8e1b8-8ddd-4298-9c61-f326e03387a7-kube-api-access-l5prb\") pod \"authorino-557c8f8584-6lfw5\" (UID: \"06f8e1b8-8ddd-4298-9c61-f326e03387a7\") " pod="kuadrant-system/authorino-557c8f8584-6lfw5"
Oct 30 03:13:53.505126 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.505101    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06f8e1b8-8ddd-4298-9c61-f326e03387a7-tls-cert\") pod \"authorino-557c8f8584-6lfw5\" (UID: \"06f8e1b8-8ddd-4298-9c61-f326e03387a7\") " pod="kuadrant-system/authorino-557c8f8584-6lfw5"
Oct 30 03:13:53.505256 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.505144    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5prb\" (UniqueName: \"kubernetes.io/projected/06f8e1b8-8ddd-4298-9c61-f326e03387a7-kube-api-access-l5prb\") pod \"authorino-557c8f8584-6lfw5\" (UID: \"06f8e1b8-8ddd-4298-9c61-f326e03387a7\") " pod="kuadrant-system/authorino-557c8f8584-6lfw5"
Oct 30 03:13:53.507717 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.507693    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/06f8e1b8-8ddd-4298-9c61-f326e03387a7-tls-cert\") pod \"authorino-557c8f8584-6lfw5\" (UID: \"06f8e1b8-8ddd-4298-9c61-f326e03387a7\") " pod="kuadrant-system/authorino-557c8f8584-6lfw5"
Oct 30 03:13:53.528467 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.528441    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5prb\" (UniqueName: \"kubernetes.io/projected/06f8e1b8-8ddd-4298-9c61-f326e03387a7-kube-api-access-l5prb\") pod \"authorino-557c8f8584-6lfw5\" (UID: \"06f8e1b8-8ddd-4298-9c61-f326e03387a7\") " pod="kuadrant-system/authorino-557c8f8584-6lfw5"
Oct 30 03:13:53.615956 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.615890    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557c8f8584-6lfw5"
Oct 30 03:13:53.904575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:53.904542    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557c8f8584-6lfw5"]
Oct 30 03:13:53.905325 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:13:53.905302    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06f8e1b8_8ddd_4298_9c61_f326e03387a7.slice/crio-b395120c16a8adeaf586026bd88bb3799e4cf71dbddd38526d8401713d0c52af WatchSource:0}: Error finding container b395120c16a8adeaf586026bd88bb3799e4cf71dbddd38526d8401713d0c52af: Status 404 returned error can't find the container with id b395120c16a8adeaf586026bd88bb3799e4cf71dbddd38526d8401713d0c52af
Oct 30 03:13:54.082944 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:13:54.082885    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:13:54.223316 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:54.223229    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557c8f8584-6lfw5" event={"ID":"06f8e1b8-8ddd-4298-9c61-f326e03387a7","Type":"ContainerStarted","Data":"b395120c16a8adeaf586026bd88bb3799e4cf71dbddd38526d8401713d0c52af"}
Oct 30 03:13:55.229298 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:55.229261    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557c8f8584-6lfw5" event={"ID":"06f8e1b8-8ddd-4298-9c61-f326e03387a7","Type":"ContainerStarted","Data":"ca6e1f02129c2e263c24ce5b8b7a355a57e495734d80c96aa66f634c06b21de7"}
Oct 30 03:13:55.268196 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:13:55.268142    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-557c8f8584-6lfw5" podStartSLOduration=1.797431105 podStartE2EDuration="2.268128301s" totalImagesPullingTime="470.697196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:55.267838886 +0000 UTC m=+1099.474019107" watchObservedRunningTime="2026-10-30 03:13:55.268128301 +0000 UTC m=+1099.474308518"
Oct 30 03:14:24.081407 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:14:24.081363    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:14:54.082223 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:14:54.082178    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:15:24.081188 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:15:24.081146    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:15:54.082591 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:15:54.082509    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:16:14.363619 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.363581    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-p6j9v"]
Oct 30 03:16:14.367036 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.367021    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-p6j9v"
Oct 30 03:16:14.387144 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.387100    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 03:16:14.387262 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.387249    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 03:16:14.399512 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.399480    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 03:16:14.399698 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.399496    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 30 03:16:14.399698 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.399523    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tk2jb\""
Oct 30 03:16:14.400886 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.400867    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-p6j9v"]
Oct 30 03:16:14.477008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.476967    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d54363-56f2-4af7-bb22-09e1d76fb315-model-cache\") pod \"s3-init-p6j9v\" (UID: \"40d54363-56f2-4af7-bb22-09e1d76fb315\") " pod="kserve/s3-init-p6j9v"
Oct 30 03:16:14.477216 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.477130    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdf7\" (UniqueName: \"kubernetes.io/projected/40d54363-56f2-4af7-bb22-09e1d76fb315-kube-api-access-skdf7\") pod \"s3-init-p6j9v\" (UID: \"40d54363-56f2-4af7-bb22-09e1d76fb315\") " pod="kserve/s3-init-p6j9v"
Oct 30 03:16:14.578309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.578272    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdf7\" (UniqueName: \"kubernetes.io/projected/40d54363-56f2-4af7-bb22-09e1d76fb315-kube-api-access-skdf7\") pod \"s3-init-p6j9v\" (UID: \"40d54363-56f2-4af7-bb22-09e1d76fb315\") " pod="kserve/s3-init-p6j9v"
Oct 30 03:16:14.578309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.578326    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d54363-56f2-4af7-bb22-09e1d76fb315-model-cache\") pod \"s3-init-p6j9v\" (UID: \"40d54363-56f2-4af7-bb22-09e1d76fb315\") " pod="kserve/s3-init-p6j9v"
Oct 30 03:16:14.578685 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.578668    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d54363-56f2-4af7-bb22-09e1d76fb315-model-cache\") pod \"s3-init-p6j9v\" (UID: \"40d54363-56f2-4af7-bb22-09e1d76fb315\") " pod="kserve/s3-init-p6j9v"
Oct 30 03:16:14.602193 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.602163    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdf7\" (UniqueName: \"kubernetes.io/projected/40d54363-56f2-4af7-bb22-09e1d76fb315-kube-api-access-skdf7\") pod \"s3-init-p6j9v\" (UID: \"40d54363-56f2-4af7-bb22-09e1d76fb315\") " pod="kserve/s3-init-p6j9v"
Oct 30 03:16:14.679818 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.679731    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-p6j9v"
Oct 30 03:16:14.983691 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.983656    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-p6j9v"]
Oct 30 03:16:14.984871 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:16:14.984844    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d54363_56f2_4af7_bb22_09e1d76fb315.slice/crio-b2d74a6dc8d0db1b3f52170c9d7ccab33306a70c06d1606b5e99511cd94b263d WatchSource:0}: Error finding container b2d74a6dc8d0db1b3f52170c9d7ccab33306a70c06d1606b5e99511cd94b263d: Status 404 returned error can't find the container with id b2d74a6dc8d0db1b3f52170c9d7ccab33306a70c06d1606b5e99511cd94b263d
Oct 30 03:16:14.987156 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:14.987141    4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:16:15.842258 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:15.842208    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p6j9v" event={"ID":"40d54363-56f2-4af7-bb22-09e1d76fb315","Type":"ContainerStarted","Data":"b2d74a6dc8d0db1b3f52170c9d7ccab33306a70c06d1606b5e99511cd94b263d"}
Oct 30 03:16:18.860999 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:18.860959    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p6j9v" event={"ID":"40d54363-56f2-4af7-bb22-09e1d76fb315","Type":"ContainerStarted","Data":"87c76708db9543bc78c1e6050ad5c323d8dc0a9ffd1a5441e46e5f127d308bf9"}
Oct 30 03:16:19.867394 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:19.867359    4494 generic.go:356] "Generic (PLEG): container finished" podID="40d54363-56f2-4af7-bb22-09e1d76fb315" containerID="87c76708db9543bc78c1e6050ad5c323d8dc0a9ffd1a5441e46e5f127d308bf9" exitCode=0
Oct 30 03:16:19.867838 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:19.867404    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p6j9v" event={"ID":"40d54363-56f2-4af7-bb22-09e1d76fb315","Type":"ContainerDied","Data":"87c76708db9543bc78c1e6050ad5c323d8dc0a9ffd1a5441e46e5f127d308bf9"}
Oct 30 03:16:24.085351 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:16:24.085313    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:16:24.894539 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:24.894502    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p6j9v" event={"ID":"40d54363-56f2-4af7-bb22-09e1d76fb315","Type":"ContainerStarted","Data":"79cac1362b32f8926b5054034ca036b95f44ce80068c924c25c98c598c65b147"}
Oct 30 03:16:24.937004 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:16:24.936948    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-p6j9v" podStartSLOduration=1.723043391 podStartE2EDuration="10.936930872s" totalImagesPullingTime="7.978263551s" totalInitContainerRuntime="1.23562393s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:24.935563666 +0000 UTC m=+1249.141743882" watchObservedRunningTime="2026-10-30 03:16:24.936930872 +0000 UTC m=+1249.143111089"
Oct 30 03:16:54.081788 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:16:54.081745    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:17:24.081762 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:17:24.081724    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:17:30.181343 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:30.181307    4494 generic.go:356] "Generic (PLEG): container finished" podID="40d54363-56f2-4af7-bb22-09e1d76fb315" containerID="79cac1362b32f8926b5054034ca036b95f44ce80068c924c25c98c598c65b147" exitCode=0
Oct 30 03:17:30.181803 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:30.181382    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p6j9v" event={"ID":"40d54363-56f2-4af7-bb22-09e1d76fb315","Type":"ContainerDied","Data":"79cac1362b32f8926b5054034ca036b95f44ce80068c924c25c98c598c65b147"}
Oct 30 03:17:31.321693 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:31.321661    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-p6j9v"
Oct 30 03:17:31.342085 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:31.342052    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d54363-56f2-4af7-bb22-09e1d76fb315-kube-api-access-skdf7\" (UniqueName: \"kubernetes.io/projected/40d54363-56f2-4af7-bb22-09e1d76fb315-kube-api-access-skdf7\") pod \"40d54363-56f2-4af7-bb22-09e1d76fb315\" (UID: \"40d54363-56f2-4af7-bb22-09e1d76fb315\") "
Oct 30 03:17:31.342254 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:31.342107    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d54363-56f2-4af7-bb22-09e1d76fb315-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d54363-56f2-4af7-bb22-09e1d76fb315-model-cache\") pod \"40d54363-56f2-4af7-bb22-09e1d76fb315\" (UID: \"40d54363-56f2-4af7-bb22-09e1d76fb315\") "
Oct 30 03:17:31.344473 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:31.344440    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d54363-56f2-4af7-bb22-09e1d76fb315-kube-api-access-skdf7" pod "40d54363-56f2-4af7-bb22-09e1d76fb315" (UID: "40d54363-56f2-4af7-bb22-09e1d76fb315"). InnerVolumeSpecName "kube-api-access-skdf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:17:31.360546 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:31.360514    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d54363-56f2-4af7-bb22-09e1d76fb315-model-cache" pod "40d54363-56f2-4af7-bb22-09e1d76fb315" (UID: "40d54363-56f2-4af7-bb22-09e1d76fb315"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:17:31.442716 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:31.442625    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skdf7\" (UniqueName: \"kubernetes.io/projected/40d54363-56f2-4af7-bb22-09e1d76fb315-kube-api-access-skdf7\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:17:31.442716 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:31.442654    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40d54363-56f2-4af7-bb22-09e1d76fb315-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:17:32.192554 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:32.192475    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-p6j9v"
Oct 30 03:17:32.192702 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:32.192472    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-p6j9v" event={"ID":"40d54363-56f2-4af7-bb22-09e1d76fb315","Type":"ContainerDied","Data":"b2d74a6dc8d0db1b3f52170c9d7ccab33306a70c06d1606b5e99511cd94b263d"}
Oct 30 03:17:32.192702 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:17:32.192588    4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2d74a6dc8d0db1b3f52170c9d7ccab33306a70c06d1606b5e99511cd94b263d"
Oct 30 03:17:54.080643 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:17:54.080602    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:18:13.984217 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:13.984178    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"]
Oct 30 03:18:13.984807 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:13.984725    4494 cpu_manager.go:414] "containerMap: removing container" podUID="40d54363-56f2-4af7-bb22-09e1d76fb315" containerName="download-hf-model"
Oct 30 03:18:13.984807 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:13.984746    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d54363-56f2-4af7-bb22-09e1d76fb315" containerName="download-hf-model"
Oct 30 03:18:13.984807 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:13.984782    4494 cpu_manager.go:414] "containerMap: removing container" podUID="40d54363-56f2-4af7-bb22-09e1d76fb315" containerName="s3-init"
Oct 30 03:18:13.984807 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:13.984790    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d54363-56f2-4af7-bb22-09e1d76fb315" containerName="s3-init"
Oct 30 03:18:13.985109 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:13.984880    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d54363-56f2-4af7-bb22-09e1d76fb315" containerName="s3-init"
Oct 30 03:18:13.988056 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:13.988032    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.020135 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.020109    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-n7dfh\""
Oct 30 03:18:14.020135 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.020109    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Oct 30 03:18:14.020135 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.020137    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Oct 30 03:18:14.020366 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.020114    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Oct 30 03:18:14.023700 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.023124    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"]
Oct 30 03:18:14.116450 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.116416    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.116632 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.116465    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.116632 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.116486    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.116632 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.116608    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.116740 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.116637    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gn9w\" (UniqueName: \"kubernetes.io/projected/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-kube-api-access-5gn9w\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.116740 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.116657    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.217469 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.217430    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.217469 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.217480    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.217728 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.217497    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.217728 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.217539    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.217728 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.217564    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gn9w\" (UniqueName: \"kubernetes.io/projected/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-kube-api-access-5gn9w\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.217728 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.217590    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.218027 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.218000    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tmp-dir\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.218099 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.218028    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-model-cache\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.218141 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.218091    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-home\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.220178 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.220158    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-dshm\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.220376 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.220358    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tls-certs\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.241360 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.241294    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gn9w\" (UniqueName: \"kubernetes.io/projected/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-kube-api-access-5gn9w\") pod \"scheduler-inline-config-test-kserve-5d95b5b968-hh9pk\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.301013 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.300978    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:14.597968 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:14.597941    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"]
Oct 30 03:18:14.599460 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:18:14.599433    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1922b111_fa51_4afc_b3aa_baa7bc3fadeb.slice/crio-65892be67c0e12cb1aa3546ce8cbcf9d3a10a84a2426edfdf273eafe57cae83c WatchSource:0}: Error finding container 65892be67c0e12cb1aa3546ce8cbcf9d3a10a84a2426edfdf273eafe57cae83c: Status 404 returned error can't find the container with id 65892be67c0e12cb1aa3546ce8cbcf9d3a10a84a2426edfdf273eafe57cae83c
Oct 30 03:18:15.374812 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:15.374763    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk" event={"ID":"1922b111-fa51-4afc-b3aa-baa7bc3fadeb","Type":"ContainerStarted","Data":"65892be67c0e12cb1aa3546ce8cbcf9d3a10a84a2426edfdf273eafe57cae83c"}
Oct 30 03:18:16.380973 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:16.380929    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk" event={"ID":"1922b111-fa51-4afc-b3aa-baa7bc3fadeb","Type":"ContainerStarted","Data":"732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958"}
Oct 30 03:18:16.426789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:16.426731    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk" podStartSLOduration=2.028045534 podStartE2EDuration="3.426713131s" totalImagesPullingTime="1.398667597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:16.426291407 +0000 UTC m=+1360.632471638" watchObservedRunningTime="2026-10-30 03:18:16.426713131 +0000 UTC m=+1360.632893348"
Oct 30 03:18:17.304064 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:17.304007    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:21.563193 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.563148    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"]
Oct 30 03:18:21.566937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.566889    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.605859 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.589519    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"default-dockercfg-fs2tf\""
Oct 30 03:18:21.605859 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.605085    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"cpu-accel-preset-kserve-self-signed-certs\""
Oct 30 03:18:21.605859 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.605230    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"kube-root-ca.crt\""
Oct 30 03:18:21.605859 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.605304    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"seaweedfs-s3-creds\""
Oct 30 03:18:21.608219 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.606173    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"openshift-service-ca.crt\""
Oct 30 03:18:21.608219 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.606791    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"]
Oct 30 03:18:21.608380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.608357    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-tmp-dir\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.608474 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.608405    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-model-cache\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.608538 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.608515    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-home\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.608609 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.608548    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-kserve-provision-location\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.608609 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.608576    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-dshm\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.608609 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.608599    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7jrh\" (UniqueName: \"kubernetes.io/projected/e04a28e3-fc56-4b50-828b-e82f314a096e-kube-api-access-f7jrh\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.608762 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.608636    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e04a28e3-fc56-4b50-828b-e82f314a096e-tls-certs\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.709385 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.709343    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-home\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.709385 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.709390    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-kserve-provision-location\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.709664 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.709408    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-dshm\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.709664 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.709424    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7jrh\" (UniqueName: \"kubernetes.io/projected/e04a28e3-fc56-4b50-828b-e82f314a096e-kube-api-access-f7jrh\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.709664 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.709446    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e04a28e3-fc56-4b50-828b-e82f314a096e-tls-certs\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.709664 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.709484    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-tmp-dir\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.709664 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.709512    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-model-cache\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.709991 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.709851    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-kserve-provision-location\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.709991 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.709941    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-model-cache\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.710057 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.709994    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-tmp-dir\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.710153 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.710134    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-home\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.712382 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.712349    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e04a28e3-fc56-4b50-828b-e82f314a096e-tls-certs\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.712847 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.712831    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-dshm\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.733967 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.733934    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7jrh\" (UniqueName: \"kubernetes.io/projected/e04a28e3-fc56-4b50-828b-e82f314a096e-kube-api-access-f7jrh\") pod \"cpu-accel-preset-kserve-84b7459d68-5q6b4\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:21.915052 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:21.914951    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:22.242598 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:22.242565    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"]
Oct 30 03:18:22.243431 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:18:22.243402    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04a28e3_fc56_4b50_828b_e82f314a096e.slice/crio-d75f4990be313c370c17c019281b7847e58f05b092b3f1472c045560e6b2ca7e WatchSource:0}: Error finding container d75f4990be313c370c17c019281b7847e58f05b092b3f1472c045560e6b2ca7e: Status 404 returned error can't find the container with id d75f4990be313c370c17c019281b7847e58f05b092b3f1472c045560e6b2ca7e
Oct 30 03:18:22.429131 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:22.429090    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4" event={"ID":"e04a28e3-fc56-4b50-828b-e82f314a096e","Type":"ContainerStarted","Data":"69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a"}
Oct 30 03:18:22.429333 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:22.429150    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4" event={"ID":"e04a28e3-fc56-4b50-828b-e82f314a096e","Type":"ContainerStarted","Data":"d75f4990be313c370c17c019281b7847e58f05b092b3f1472c045560e6b2ca7e"}
Oct 30 03:18:24.085632 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:18:24.085583    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:18:24.302129 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:24.302088    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:24.315268 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:24.315239    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:24.453186 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:24.453101    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:25.281147 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:25.281111    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"]
Oct 30 03:18:25.281637 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:25.281293    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4" podUID="e04a28e3-fc56-4b50-828b-e82f314a096e" containerName="storage-initializer" containerID="cri-o://69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a" gracePeriod=60
Oct 30 03:18:25.448046 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:25.448007    4494 generic.go:356] "Generic (PLEG): container finished" podID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerID="951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f" exitCode=0
Oct 30 03:18:25.448254 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:25.448085    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" event={"ID":"92a3121c-9c1e-4085-a4b6-151ccd0438e5","Type":"ContainerDied","Data":"951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f"}
Oct 30 03:18:25.448559 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:25.448540    4494 scope.go:122] "RemoveContainer" containerID="951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f"
Oct 30 03:18:25.449658 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:25.449629    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:18:25.449745 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:25.449700    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:18:25.449745 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:25.449735    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:18:26.455498 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:26.455466    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" event={"ID":"92a3121c-9c1e-4085-a4b6-151ccd0438e5","Type":"ContainerStarted","Data":"27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db"}
Oct 30 03:18:26.795937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:26.795349    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:18:26.795937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:26.795487    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 30 03:18:26.795937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:26.795532    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 30 03:18:27.801926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:27.799050    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 30 03:18:27.801926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:27.799235    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 30 03:18:28.160284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.160260    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:28.275336 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.275302    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04a28e3-fc56-4b50-828b-e82f314a096e-kube-api-access-f7jrh\" (UniqueName: \"kubernetes.io/projected/e04a28e3-fc56-4b50-828b-e82f314a096e-kube-api-access-f7jrh\") pod \"e04a28e3-fc56-4b50-828b-e82f314a096e\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") "
Oct 30 03:18:28.275523 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.275382    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-dshm\") pod \"e04a28e3-fc56-4b50-828b-e82f314a096e\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") "
Oct 30 03:18:28.275523 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.275411    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e04a28e3-fc56-4b50-828b-e82f314a096e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e04a28e3-fc56-4b50-828b-e82f314a096e-tls-certs\") pod \"e04a28e3-fc56-4b50-828b-e82f314a096e\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") "
Oct 30 03:18:28.275523 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.275432    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-home\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-home\") pod \"e04a28e3-fc56-4b50-828b-e82f314a096e\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") "
Oct 30 03:18:28.275523 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.275481    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-model-cache\") pod \"e04a28e3-fc56-4b50-828b-e82f314a096e\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") "
Oct 30 03:18:28.275523 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.275514    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-tmp-dir\") pod \"e04a28e3-fc56-4b50-828b-e82f314a096e\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") "
Oct 30 03:18:28.275709 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.275546    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-kserve-provision-location\") pod \"e04a28e3-fc56-4b50-828b-e82f314a096e\" (UID: \"e04a28e3-fc56-4b50-828b-e82f314a096e\") "
Oct 30 03:18:28.275745 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.275712    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-home" pod "e04a28e3-fc56-4b50-828b-e82f314a096e" (UID: "e04a28e3-fc56-4b50-828b-e82f314a096e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:28.275933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.275887    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:28.276100 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.275937    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-model-cache" pod "e04a28e3-fc56-4b50-828b-e82f314a096e" (UID: "e04a28e3-fc56-4b50-828b-e82f314a096e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:28.276100 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.276074    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-tmp-dir" pod "e04a28e3-fc56-4b50-828b-e82f314a096e" (UID: "e04a28e3-fc56-4b50-828b-e82f314a096e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:28.277934 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.277889    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e04a28e3-fc56-4b50-828b-e82f314a096e-tls-certs" pod "e04a28e3-fc56-4b50-828b-e82f314a096e" (UID: "e04a28e3-fc56-4b50-828b-e82f314a096e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:28.278030 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.277976    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-dshm" pod "e04a28e3-fc56-4b50-828b-e82f314a096e" (UID: "e04a28e3-fc56-4b50-828b-e82f314a096e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:28.278320 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.278296    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04a28e3-fc56-4b50-828b-e82f314a096e-kube-api-access-f7jrh" pod "e04a28e3-fc56-4b50-828b-e82f314a096e" (UID: "e04a28e3-fc56-4b50-828b-e82f314a096e"). InnerVolumeSpecName "kube-api-access-f7jrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:28.376819 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.376743    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:28.376819 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.376772    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:28.376819 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.376783    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7jrh\" (UniqueName: \"kubernetes.io/projected/e04a28e3-fc56-4b50-828b-e82f314a096e-kube-api-access-f7jrh\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:28.376819 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.376802    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:28.376819 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.376810    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e04a28e3-fc56-4b50-828b-e82f314a096e-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:28.457525 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.457480    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-kserve-provision-location" pod "e04a28e3-fc56-4b50-828b-e82f314a096e" (UID: "e04a28e3-fc56-4b50-828b-e82f314a096e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:28.466347 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.466309    4494 generic.go:356] "Generic (PLEG): container finished" podID="e04a28e3-fc56-4b50-828b-e82f314a096e" containerID="69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a" exitCode=0
Oct 30 03:18:28.466499 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.466360    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"
Oct 30 03:18:28.466499 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.466382    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4" event={"ID":"e04a28e3-fc56-4b50-828b-e82f314a096e","Type":"ContainerDied","Data":"69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a"}
Oct 30 03:18:28.466499 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.466421    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4" event={"ID":"e04a28e3-fc56-4b50-828b-e82f314a096e","Type":"ContainerDied","Data":"d75f4990be313c370c17c019281b7847e58f05b092b3f1472c045560e6b2ca7e"}
Oct 30 03:18:28.466499 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.466441    4494 scope.go:122] "RemoveContainer" containerID="69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a"
Oct 30 03:18:28.476503 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.476479    4494 scope.go:122] "RemoveContainer" containerID="69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a"
Oct 30 03:18:28.476770 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:18:28.476751    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a\": container with ID starting with 69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a not found: ID does not exist" containerID="69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a"
Oct 30 03:18:28.476840 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.476779    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a"} err="failed to get container status \"69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a\": rpc error: code = NotFound desc = could not find container \"69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a\": container with ID starting with 69489e857181c101944ebd8e34ed3036293a53d19d5c86f3c94f1195d69d927a not found: ID does not exist"
Oct 30 03:18:28.478260 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.478238    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e04a28e3-fc56-4b50-828b-e82f314a096e-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:28.561122 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.561091    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"]
Oct 30 03:18:28.566058 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.566035    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-84b7459d68-5q6b4"]
Oct 30 03:18:28.798926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.796793    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 30 03:18:28.798926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:28.796865    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 30 03:18:29.797383 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:29.797341    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 30 03:18:29.798428 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:29.798010    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 30 03:18:29.927424 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:29.927152    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04a28e3-fc56-4b50-828b-e82f314a096e" path="/var/lib/kubelet/pods/e04a28e3-fc56-4b50-828b-e82f314a096e/volumes"
Oct 30 03:18:30.795353 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:30.795313    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Oct 30 03:18:30.795673 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:30.795640    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Oct 30 03:18:31.419040 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.419003    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"]
Oct 30 03:18:31.419462 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.419432    4494 cpu_manager.go:414] "containerMap: removing container" podUID="e04a28e3-fc56-4b50-828b-e82f314a096e" containerName="storage-initializer"
Oct 30 03:18:31.419462 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.419451    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04a28e3-fc56-4b50-828b-e82f314a096e" containerName="storage-initializer"
Oct 30 03:18:31.419614 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.419595    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04a28e3-fc56-4b50-828b-e82f314a096e" containerName="storage-initializer"
Oct 30 03:18:31.428297 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.428271    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.444541 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.444502    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"]
Oct 30 03:18:31.453842 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.453809    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"kube-root-ca.crt\""
Oct 30 03:18:31.454465 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.454440    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"openshift-service-ca.crt\""
Oct 30 03:18:31.458733 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.458709    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"seaweedfs-s3-creds\""
Oct 30 03:18:31.458872 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.458787    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"default-dockercfg-8chkv\""
Oct 30 03:18:31.458872 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.458787    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"cpu-accel-precedence-kserve-self-signed-certs\""
Oct 30 03:18:31.525005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.524972    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tls-certs\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.525199 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.525024    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-model-cache\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.525199 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.525045    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-dshm\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.525303 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.525271    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-home\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.525416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.525387    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhz72\" (UniqueName: \"kubernetes.io/projected/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kube-api-access-bhz72\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.525524 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.525499    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tmp-dir\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.525593 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.525570    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626278 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626225    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tmp-dir\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626278 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626282    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626550 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626315    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tls-certs\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626550 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626346    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-model-cache\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626550 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626364    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-dshm\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626550 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626408    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-home\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626550 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626462    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhz72\" (UniqueName: \"kubernetes.io/projected/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kube-api-access-bhz72\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626804 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626718    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tmp-dir\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626804 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626743    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-model-cache\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626804 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626775    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.626993 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.626841    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-home\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.628875 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.628846    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-dshm\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.629085 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.629069    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tls-certs\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.649990 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.649961    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhz72\" (UniqueName: \"kubernetes.io/projected/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kube-api-access-bhz72\") pod \"cpu-accel-precedence-kserve-65fdb67cbb-rr9xk\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.745556 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.745478    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:31.798553 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.798278    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:18:31.799658 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.799285    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:18:31.800024 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:31.800002    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:18:32.056813 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:32.056791    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"]
Oct 30 03:18:32.058523 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:18:32.058498    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9d43d0_9282_42ff_b9ad_19e6ffc14392.slice/crio-f5b6cf6bdd5083a82e26bc90d4e0ae2c9afbb7bf27b18a8af6244ec6930c2ae5 WatchSource:0}: Error finding container f5b6cf6bdd5083a82e26bc90d4e0ae2c9afbb7bf27b18a8af6244ec6930c2ae5: Status 404 returned error can't find the container with id f5b6cf6bdd5083a82e26bc90d4e0ae2c9afbb7bf27b18a8af6244ec6930c2ae5
Oct 30 03:18:32.487873 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:32.487782    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk" event={"ID":"2b9d43d0-9282-42ff-b9ad-19e6ffc14392","Type":"ContainerStarted","Data":"220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35"}
Oct 30 03:18:32.487873 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:32.487835    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk" event={"ID":"2b9d43d0-9282-42ff-b9ad-19e6ffc14392","Type":"ContainerStarted","Data":"f5b6cf6bdd5083a82e26bc90d4e0ae2c9afbb7bf27b18a8af6244ec6930c2ae5"}
Oct 30 03:18:34.784587 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:34.784555    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"]
Oct 30 03:18:34.785054 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:34.784764    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk" podUID="2b9d43d0-9282-42ff-b9ad-19e6ffc14392" containerName="storage-initializer" containerID="cri-o://220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35" gracePeriod=60
Oct 30 03:18:36.935181 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:36.935156    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:37.081577 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.081536    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kserve-provision-location\") pod \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") "
Oct 30 03:18:37.081785 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.081625    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tls-certs\") pod \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") "
Oct 30 03:18:37.081785 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.081709    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-dshm\") pod \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") "
Oct 30 03:18:37.081785 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.081738    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kube-api-access-bhz72\" (UniqueName: \"kubernetes.io/projected/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kube-api-access-bhz72\") pod \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") "
Oct 30 03:18:37.081785 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.081762    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tmp-dir\") pod \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") "
Oct 30 03:18:37.082027 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.081794    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-model-cache\") pod \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") "
Oct 30 03:18:37.082027 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.081855    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-home\") pod \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\" (UID: \"2b9d43d0-9282-42ff-b9ad-19e6ffc14392\") "
Oct 30 03:18:37.082150 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.082095    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tmp-dir" pod "2b9d43d0-9282-42ff-b9ad-19e6ffc14392" (UID: "2b9d43d0-9282-42ff-b9ad-19e6ffc14392"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:37.082195 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.082158    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-model-cache" pod "2b9d43d0-9282-42ff-b9ad-19e6ffc14392" (UID: "2b9d43d0-9282-42ff-b9ad-19e6ffc14392"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:37.082231 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.082190    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-home" pod "2b9d43d0-9282-42ff-b9ad-19e6ffc14392" (UID: "2b9d43d0-9282-42ff-b9ad-19e6ffc14392"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:37.084184 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.084153    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tls-certs" pod "2b9d43d0-9282-42ff-b9ad-19e6ffc14392" (UID: "2b9d43d0-9282-42ff-b9ad-19e6ffc14392"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:37.084298 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.084187    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kube-api-access-bhz72" pod "2b9d43d0-9282-42ff-b9ad-19e6ffc14392" (UID: "2b9d43d0-9282-42ff-b9ad-19e6ffc14392"). InnerVolumeSpecName "kube-api-access-bhz72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:37.084298 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.084270    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-dshm" pod "2b9d43d0-9282-42ff-b9ad-19e6ffc14392" (UID: "2b9d43d0-9282-42ff-b9ad-19e6ffc14392"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:37.148006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.147956    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kserve-provision-location" pod "2b9d43d0-9282-42ff-b9ad-19e6ffc14392" (UID: "2b9d43d0-9282-42ff-b9ad-19e6ffc14392"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:37.183008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.182965    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:37.183008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.182999    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhz72\" (UniqueName: \"kubernetes.io/projected/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kube-api-access-bhz72\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:37.183238 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.183036    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:37.183238 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.183049    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:37.183238 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.183060    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:37.183238 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.183070    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:37.183238 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.183081    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b9d43d0-9282-42ff-b9ad-19e6ffc14392-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:37.512148 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.512062    4494 generic.go:356] "Generic (PLEG): container finished" podID="2b9d43d0-9282-42ff-b9ad-19e6ffc14392" containerID="220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35" exitCode=0
Oct 30 03:18:37.512148 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.512112    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"
Oct 30 03:18:37.512356 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.512150    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk" event={"ID":"2b9d43d0-9282-42ff-b9ad-19e6ffc14392","Type":"ContainerDied","Data":"220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35"}
Oct 30 03:18:37.512356 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.512191    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk" event={"ID":"2b9d43d0-9282-42ff-b9ad-19e6ffc14392","Type":"ContainerDied","Data":"f5b6cf6bdd5083a82e26bc90d4e0ae2c9afbb7bf27b18a8af6244ec6930c2ae5"}
Oct 30 03:18:37.512356 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.512208    4494 scope.go:122] "RemoveContainer" containerID="220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35"
Oct 30 03:18:37.521845 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.521826    4494 scope.go:122] "RemoveContainer" containerID="220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35"
Oct 30 03:18:37.522140 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:18:37.522119    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35\": container with ID starting with 220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35 not found: ID does not exist" containerID="220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35"
Oct 30 03:18:37.522211 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.522155    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35"} err="failed to get container status \"220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35\": rpc error: code = NotFound desc = could not find container \"220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35\": container with ID starting with 220ce10a9551a57ed2c8f6d34a8e08a160ad2f0d0f99fea99229c5f3c06a9a35 not found: ID does not exist"
Oct 30 03:18:37.595589 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.595552    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"]
Oct 30 03:18:37.601512 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.601478    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-65fdb67cbb-rr9xk"]
Oct 30 03:18:37.926752 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:37.926707    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9d43d0-9282-42ff-b9ad-19e6ffc14392" path="/var/lib/kubelet/pods/2b9d43d0-9282-42ff-b9ad-19e6ffc14392/volumes"
Oct 30 03:18:49.692257 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.692183    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"]
Oct 30 03:18:49.692611 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.692590    4494 cpu_manager.go:414] "containerMap: removing container" podUID="2b9d43d0-9282-42ff-b9ad-19e6ffc14392" containerName="storage-initializer"
Oct 30 03:18:49.692611 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.692603    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9d43d0-9282-42ff-b9ad-19e6ffc14392" containerName="storage-initializer"
Oct 30 03:18:49.692706 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.692662    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9d43d0-9282-42ff-b9ad-19e6ffc14392" containerName="storage-initializer"
Oct 30 03:18:49.697599 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.697578    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.712504 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.712478    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"]
Oct 30 03:18:49.722916 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.722834    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Oct 30 03:18:49.722916 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.722834    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:18:49.722916 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.722862    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Oct 30 03:18:49.722916 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.722877    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-7wlnm\""
Oct 30 03:18:49.723148 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.722916    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"seaweedfs-s3-creds\""
Oct 30 03:18:49.723148 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.722958    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Oct 30 03:18:49.778534 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.778495    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbv8n\" (UniqueName: \"kubernetes.io/projected/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kube-api-access-xbv8n\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.778683 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.778541    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.778683 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.778566    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.778683 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.778665    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08ee3e3c-edfb-4820-b8fa-71387bcb490d-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.778811 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.778736    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.778811 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.778755    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.778811 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.778786    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.778933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.778837    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880116 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880073    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880116 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880118    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880358 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880144    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08ee3e3c-edfb-4820-b8fa-71387bcb490d-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880358 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880176    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880358 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880193    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880358 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880212    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880358 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880228    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880358 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880275    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbv8n\" (UniqueName: \"kubernetes.io/projected/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kube-api-access-xbv8n\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880672 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880569    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880672 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880595    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880672 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880633    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.880855 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.880835    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.881030 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.881010    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08ee3e3c-edfb-4820-b8fa-71387bcb490d-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.882733 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.882715    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.883026 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.883004    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:49.905517 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:49.905480    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbv8n\" (UniqueName: \"kubernetes.io/projected/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kube-api-access-xbv8n\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:50.018722 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:50.018639    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:18:50.352365 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:50.352332    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"]
Oct 30 03:18:50.354053 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:18:50.354029    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ee3e3c_edfb_4820_b8fa_71387bcb490d.slice/crio-1356fff69f0e2a2a5c3f040654f48dd1ee48832603358278d70bf6cba0fd901d WatchSource:0}: Error finding container 1356fff69f0e2a2a5c3f040654f48dd1ee48832603358278d70bf6cba0fd901d: Status 404 returned error can't find the container with id 1356fff69f0e2a2a5c3f040654f48dd1ee48832603358278d70bf6cba0fd901d
Oct 30 03:18:50.570449 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:50.570405    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" event={"ID":"08ee3e3c-edfb-4820-b8fa-71387bcb490d","Type":"ContainerStarted","Data":"e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4"}
Oct 30 03:18:50.570629 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:50.570464    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" event={"ID":"08ee3e3c-edfb-4820-b8fa-71387bcb490d","Type":"ContainerStarted","Data":"1356fff69f0e2a2a5c3f040654f48dd1ee48832603358278d70bf6cba0fd901d"}
Oct 30 03:18:51.690829 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:51.690780    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"]
Oct 30 03:18:51.705140 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:18:51.705093    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958" cmd=["/bin/sleep","15"]
Oct 30 03:18:51.705337 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:51.705165    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:18:51.705337 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:18:51.705178    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk" podUID="1922b111-fa51-4afc-b3aa-baa7bc3fadeb" containerName="main" containerID="cri-o://732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958"
Oct 30 03:18:51.705337 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:51.705214    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk" podUID="1922b111-fa51-4afc-b3aa-baa7bc3fadeb" containerName="main" containerID="cri-o://732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958" gracePeriod=60
Oct 30 03:18:52.086736 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.086703    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:52.103267 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.103231    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tmp-dir\") pod \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") "
Oct 30 03:18:52.103474 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.103354    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tls-certs\" (UniqueName: \"kubernetes.io/secret/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tls-certs\") pod \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") "
Oct 30 03:18:52.103474 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.103385    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-home\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-home\") pod \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") "
Oct 30 03:18:52.103474 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.103421    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-model-cache\") pod \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") "
Oct 30 03:18:52.103474 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.103451    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-kube-api-access-5gn9w\" (UniqueName: \"kubernetes.io/projected/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-kube-api-access-5gn9w\") pod \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") "
Oct 30 03:18:52.103631 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.103494    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-dshm\") pod \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\" (UID: \"1922b111-fa51-4afc-b3aa-baa7bc3fadeb\") "
Oct 30 03:18:52.103631 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.103503    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tmp-dir" pod "1922b111-fa51-4afc-b3aa-baa7bc3fadeb" (UID: "1922b111-fa51-4afc-b3aa-baa7bc3fadeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:52.103699 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.103638    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-home" pod "1922b111-fa51-4afc-b3aa-baa7bc3fadeb" (UID: "1922b111-fa51-4afc-b3aa-baa7bc3fadeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:52.103804 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.103784    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:52.103880 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.103810    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:52.104695 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.104105    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-model-cache" pod "1922b111-fa51-4afc-b3aa-baa7bc3fadeb" (UID: "1922b111-fa51-4afc-b3aa-baa7bc3fadeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:52.108346 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.106495    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-dshm" pod "1922b111-fa51-4afc-b3aa-baa7bc3fadeb" (UID: "1922b111-fa51-4afc-b3aa-baa7bc3fadeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:18:52.108346 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.106691    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tls-certs" pod "1922b111-fa51-4afc-b3aa-baa7bc3fadeb" (UID: "1922b111-fa51-4afc-b3aa-baa7bc3fadeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:18:52.108346 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.107028    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-kube-api-access-5gn9w" pod "1922b111-fa51-4afc-b3aa-baa7bc3fadeb" (UID: "1922b111-fa51-4afc-b3aa-baa7bc3fadeb"). InnerVolumeSpecName "kube-api-access-5gn9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:18:52.207927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.207623    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:52.207927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.207670    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:52.207927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.207686    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gn9w\" (UniqueName: \"kubernetes.io/projected/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-kube-api-access-5gn9w\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:52.207927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.207702    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1922b111-fa51-4afc-b3aa-baa7bc3fadeb-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:18:52.580912 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.580861    4494 generic.go:356] "Generic (PLEG): container finished" podID="1922b111-fa51-4afc-b3aa-baa7bc3fadeb" containerID="732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958" exitCode=0
Oct 30 03:18:52.581194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.580940    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"
Oct 30 03:18:52.581194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.580944    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk" event={"ID":"1922b111-fa51-4afc-b3aa-baa7bc3fadeb","Type":"ContainerDied","Data":"732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958"}
Oct 30 03:18:52.581194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.581010    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk" event={"ID":"1922b111-fa51-4afc-b3aa-baa7bc3fadeb","Type":"ContainerDied","Data":"65892be67c0e12cb1aa3546ce8cbcf9d3a10a84a2426edfdf273eafe57cae83c"}
Oct 30 03:18:52.581194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.581040    4494 scope.go:122] "RemoveContainer" containerID="732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958"
Oct 30 03:18:52.590026 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.589883    4494 scope.go:122] "RemoveContainer" containerID="732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958"
Oct 30 03:18:52.590304 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:18:52.590278    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958\": container with ID starting with 732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958 not found: ID does not exist" containerID="732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958"
Oct 30 03:18:52.590391 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.590319    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958"} err="failed to get container status \"732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958\": rpc error: code = NotFound desc = could not find container \"732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958\": container with ID starting with 732ec92bc0138949d0c1ce04910699e9f782909753e867d5e63d06de57eed958 not found: ID does not exist"
Oct 30 03:18:52.641328 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.641289    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"]
Oct 30 03:18:52.652929 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:52.652884    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-5d95b5b968-hh9pk"]
Oct 30 03:18:53.586519 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:53.586487    4494 generic.go:356] "Generic (PLEG): container finished" podID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerID="e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4" exitCode=0
Oct 30 03:18:53.587014 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:53.586573    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" event={"ID":"08ee3e3c-edfb-4820-b8fa-71387bcb490d","Type":"ContainerDied","Data":"e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4"}
Oct 30 03:18:53.927174 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:53.927101    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1922b111-fa51-4afc-b3aa-baa7bc3fadeb" path="/var/lib/kubelet/pods/1922b111-fa51-4afc-b3aa-baa7bc3fadeb/volumes"
Oct 30 03:18:54.081491 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:18:54.081450    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:18:58.682434 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.680566    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"]
Oct 30 03:18:58.682434 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.681172    4494 cpu_manager.go:414] "containerMap: removing container" podUID="1922b111-fa51-4afc-b3aa-baa7bc3fadeb" containerName="main"
Oct 30 03:18:58.682434 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.681190    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1922b111-fa51-4afc-b3aa-baa7bc3fadeb" containerName="main"
Oct 30 03:18:58.682434 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.681284    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="1922b111-fa51-4afc-b3aa-baa7bc3fadeb" containerName="main"
Oct 30 03:18:58.684311 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.684290    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.709181 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.709099    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"]
Oct 30 03:18:58.771863 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.771809    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.771863 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.771858    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66tf\" (UniqueName: \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-kube-api-access-f66tf\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.772149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.771982    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.772149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.772029    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.772149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.772122    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.772149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.772144    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.772435 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.772167    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.772435 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.772188    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41414588-b1f1-4bbe-bc40-552dd3ad266c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.772435 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.772385    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.873765 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.873717    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874011 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.873777    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41414588-b1f1-4bbe-bc40-552dd3ad266c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874011 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.873813    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874011 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.873857    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874011 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.873876    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66tf\" (UniqueName: \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-kube-api-access-f66tf\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874011 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.873926    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874011 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.873958    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874350 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.874028    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874350 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.874056    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874350 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.874207    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874350 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.874338    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.874678 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.874658    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.875211 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.875187    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.875303 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.875208    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41414588-b1f1-4bbe-bc40-552dd3ad266c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.878512 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.878479    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.887415 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.887392    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.898747 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.898714    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66tf\" (UniqueName: \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-kube-api-access-f66tf\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:58.899039 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:58.899016    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:59.002118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:59.002036    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:18:59.337560 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:59.337527    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"]
Oct 30 03:18:59.341198 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:18:59.341161    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41414588_b1f1_4bbe_bc40_552dd3ad266c.slice/crio-8fe91018f1e13e7b5d4745b58ad7a7a869a0dd94dbbe7ce609beb45131b9793c WatchSource:0}: Error finding container 8fe91018f1e13e7b5d4745b58ad7a7a869a0dd94dbbe7ce609beb45131b9793c: Status 404 returned error can't find the container with id 8fe91018f1e13e7b5d4745b58ad7a7a869a0dd94dbbe7ce609beb45131b9793c
Oct 30 03:18:59.343980 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:59.343944    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:18:59.344089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:59.344038    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:18:59.344150 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:59.344114    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:18:59.625484 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:59.625400    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" event={"ID":"41414588-b1f1-4bbe-bc40-552dd3ad266c","Type":"ContainerStarted","Data":"5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039"}
Oct 30 03:18:59.625484 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:59.625464    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" event={"ID":"41414588-b1f1-4bbe-bc40-552dd3ad266c","Type":"ContainerStarted","Data":"8fe91018f1e13e7b5d4745b58ad7a7a869a0dd94dbbe7ce609beb45131b9793c"}
Oct 30 03:18:59.678926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:18:59.678846    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" podStartSLOduration=1.678829591 podStartE2EDuration="1.678829591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:59.678358335 +0000 UTC m=+1403.884538586" watchObservedRunningTime="2026-10-30 03:18:59.678829591 +0000 UTC m=+1403.885009900"
Oct 30 03:19:00.005404 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:00.005314    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:19:00.006558 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:00.006529    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:19:00.006631 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:00.006590    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 03:19:01.003927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:01.003329    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:19:01.003927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:01.003391    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 03:19:02.005110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:02.005073    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:19:02.005523 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:02.005135    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 03:19:03.004926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:03.004284    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:19:03.004926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:03.004336    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 03:19:04.004379 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:04.004339    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:19:04.004885 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:04.004395    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 03:19:05.004062 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:05.004005    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 30 03:19:05.004265 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:05.004058    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 30 03:19:06.009806 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:06.009770    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:19:06.011069 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:06.011039    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:19:06.011960 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:06.011934    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:19:13.735334 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:13.735293    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"]
Oct 30 03:19:14.239620 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.239579    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"]
Oct 30 03:19:14.239836 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.239706    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.349671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.349606    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znf72\" (UniqueName: \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-kube-api-access-znf72\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.349851 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.349745    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.349851 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.349784    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.349851 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.349816    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51422a6f-9df9-4283-815e-a0dd6660cb3f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.349851 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.349834    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.350066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.349876    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.350066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.349892    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.350066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.350030    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.350185 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.350084    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.450797 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.450755    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451019 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.450825    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451019 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.450894    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znf72\" (UniqueName: \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-kube-api-access-znf72\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451019 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.450946    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451019 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.450968    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451019 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.450995    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51422a6f-9df9-4283-815e-a0dd6660cb3f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451280 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.451018    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451280 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.451077    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451280 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.451102    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451440 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.451277    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451440 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.451427    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451559 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.451541    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.451798 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.451766    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.452020 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.451980    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51422a6f-9df9-4283-815e-a0dd6660cb3f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.454284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.454262    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.454999 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.454975    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.477309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.477278    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.477665 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.477642    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znf72\" (UniqueName: \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-kube-api-access-znf72\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.605947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.605893    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:14.952465 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.952435    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"]
Oct 30 03:19:14.953114 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:19:14.953079    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51422a6f_9df9_4283_815e_a0dd6660cb3f.slice/crio-b7c1d1c4b1810c10ce103b894c9c3b1a0f8d97d2237f390d1265f634e4809473 WatchSource:0}: Error finding container b7c1d1c4b1810c10ce103b894c9c3b1a0f8d97d2237f390d1265f634e4809473: Status 404 returned error can't find the container with id b7c1d1c4b1810c10ce103b894c9c3b1a0f8d97d2237f390d1265f634e4809473
Oct 30 03:19:14.955740 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.955706    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:19:14.955829 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.955794    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:19:14.955876 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:14.955838    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:19:15.732347 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:15.732314    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" event={"ID":"51422a6f-9df9-4283-815e-a0dd6660cb3f","Type":"ContainerStarted","Data":"46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99"}
Oct 30 03:19:15.732539 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:15.732369    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" event={"ID":"51422a6f-9df9-4283-815e-a0dd6660cb3f","Type":"ContainerStarted","Data":"b7c1d1c4b1810c10ce103b894c9c3b1a0f8d97d2237f390d1265f634e4809473"}
Oct 30 03:19:15.818150 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:15.818096    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" podStartSLOduration=2.818083376 podStartE2EDuration="2.818083376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:15.805411006 +0000 UTC m=+1420.011591235" watchObservedRunningTime="2026-10-30 03:19:15.818083376 +0000 UTC m=+1420.024263594"
Oct 30 03:19:16.606862 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:16.606822    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:16.608107 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:16.608074    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 03:19:16.608257 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:16.608131    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 03:19:17.607060 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:17.607017    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 03:19:17.607463 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:17.607085    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 03:19:18.606305 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:18.606263    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 03:19:18.606498 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:18.606326    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 03:19:19.606450 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:19.606411    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 03:19:19.607098 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:19.606480    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 03:19:20.606954 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:20.606884    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 03:19:20.607515 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:20.606978    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 03:19:21.610986 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:21.610944    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:21.611401 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:21.611360    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:21.612086 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:21.612063    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:19:24.086791 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:19:24.086748    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:19:28.778271 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.778224    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"]
Oct 30 03:19:28.840627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.840546    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"]
Oct 30 03:19:28.840866 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.840682    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:28.921951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.916209    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:28.921951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.916281    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjnq\" (UniqueName: \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-kube-api-access-8fjnq\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:28.921951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.916326    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dda10f8-5171-473e-8e69-4c104201f3ee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:28.921951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.916378    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:28.921951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.916409    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:28.921951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.916438    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:28.921951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.916478    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:28.921951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.916525    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:28.921951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:28.916566    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.017672 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.017628    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dda10f8-5171-473e-8e69-4c104201f3ee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.017870 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.017711    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.017870 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.017738    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.017870 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.017764    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.017870 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.017803    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.017870 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.017851    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.018130 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.017890    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.018130 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.018010    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.018130 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.018059    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjnq\" (UniqueName: \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-kube-api-access-8fjnq\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.018347 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.018323    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.018578 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.018550    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.018772 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.018746    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.018988 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.018929    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dda10f8-5171-473e-8e69-4c104201f3ee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.019172 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.019149    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.021287 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.021266    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.022038 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.021999    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.041979 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.041892    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.042129 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.042073    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjnq\" (UniqueName: \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-kube-api-access-8fjnq\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.166109 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.166072    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:29.575143 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.575105    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"]
Oct 30 03:19:29.575821 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:19:29.575787    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dda10f8_5171_473e_8e69_4c104201f3ee.slice/crio-76f0dd124165ff8def03c0c9cbeebcaf915482326a0bb64ae8153c689a103b4b WatchSource:0}: Error finding container 76f0dd124165ff8def03c0c9cbeebcaf915482326a0bb64ae8153c689a103b4b: Status 404 returned error can't find the container with id 76f0dd124165ff8def03c0c9cbeebcaf915482326a0bb64ae8153c689a103b4b
Oct 30 03:19:29.578274 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.578239    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:19:29.578376 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.578320    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:19:29.578376 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.578352    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:19:29.811155 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.811044    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" event={"ID":"2dda10f8-5171-473e-8e69-4c104201f3ee","Type":"ContainerStarted","Data":"caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add"}
Oct 30 03:19:29.811155 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:29.811116    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" event={"ID":"2dda10f8-5171-473e-8e69-4c104201f3ee","Type":"ContainerStarted","Data":"76f0dd124165ff8def03c0c9cbeebcaf915482326a0bb64ae8153c689a103b4b"}
Oct 30 03:19:30.167302 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:30.167217    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:30.169284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:30.169249    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 03:19:30.169437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:30.169318    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 03:19:31.166604 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:31.166565    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 03:19:31.167077 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:31.166638    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 03:19:32.166691 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:32.166651    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 03:19:32.167155 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:32.166727    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 03:19:33.166774 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:33.166735    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 03:19:33.167246 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:33.166801    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 03:19:34.167305 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:34.167262    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 03:19:34.167761 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:34.167326    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 03:19:35.167599 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:35.167151    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 03:19:35.167599 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:35.167217    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 03:19:36.171981 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:36.171944    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:36.172430 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:36.172231    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:36.173209 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:36.173186    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:19:36.230645 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:36.230579    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" podStartSLOduration=8.230558373 podStartE2EDuration="8.230558373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:29.858013781 +0000 UTC m=+1434.064194002" watchObservedRunningTime="2026-10-30 03:19:36.230558373 +0000 UTC m=+1440.436738594"
Oct 30 03:19:43.126425 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.124615    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"]
Oct 30 03:19:43.341210 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.341173    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"]
Oct 30 03:19:43.341402 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.341255    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.382020 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.381878    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Oct 30 03:19:43.382020 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.381876    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Oct 30 03:19:43.382246 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.382051    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-z9vmd\""
Oct 30 03:19:43.382393 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.382356    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Oct 30 03:19:43.496081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.496035    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.496269 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.496089    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.496269 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.496205    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.496269 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.496230    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.496269 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.496258    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad6f1c3c-178b-4813-b19f-777a3b49532c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.496451 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.496318    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d96fc\" (UniqueName: \"kubernetes.io/projected/ad6f1c3c-178b-4813-b19f-777a3b49532c-kube-api-access-d96fc\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.597435 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.597397    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.597435 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.597442    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.597692 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.597500    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.597692 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.597522    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.597692 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.597537    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad6f1c3c-178b-4813-b19f-777a3b49532c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.597692 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.597578    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d96fc\" (UniqueName: \"kubernetes.io/projected/ad6f1c3c-178b-4813-b19f-777a3b49532c-kube-api-access-d96fc\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.599675 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.598401    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.599675 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.598672    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.599675 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.598882    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.614980 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.606241    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.614980 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.607582    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad6f1c3c-178b-4813-b19f-777a3b49532c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.622927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.622873    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d96fc\" (UniqueName: \"kubernetes.io/projected/ad6f1c3c-178b-4813-b19f-777a3b49532c-kube-api-access-d96fc\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:43.672278 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:43.672178    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:44.038105 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:44.038073    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"]
Oct 30 03:19:44.039924 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:19:44.039874    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6f1c3c_178b_4813_b19f_777a3b49532c.slice/crio-f7d7e4fb59b28846f4fe28f2208ba7b0a2b1719d8ca9cea7041403c1ffabeffd WatchSource:0}: Error finding container f7d7e4fb59b28846f4fe28f2208ba7b0a2b1719d8ca9cea7041403c1ffabeffd: Status 404 returned error can't find the container with id f7d7e4fb59b28846f4fe28f2208ba7b0a2b1719d8ca9cea7041403c1ffabeffd
Oct 30 03:19:44.891376 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:44.891335    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls" event={"ID":"ad6f1c3c-178b-4813-b19f-777a3b49532c","Type":"ContainerStarted","Data":"8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2"}
Oct 30 03:19:44.891762 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:44.891399    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls" event={"ID":"ad6f1c3c-178b-4813-b19f-777a3b49532c","Type":"ContainerStarted","Data":"f7d7e4fb59b28846f4fe28f2208ba7b0a2b1719d8ca9cea7041403c1ffabeffd"}
Oct 30 03:19:44.936015 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:44.935957    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls" podStartSLOduration=1.935939539 podStartE2EDuration="1.935939539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:44.933400129 +0000 UTC m=+1449.139580348" watchObservedRunningTime="2026-10-30 03:19:44.935939539 +0000 UTC m=+1449.142119757"
Oct 30 03:19:45.672432 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:45.672364    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:45.907413 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:45.907368    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" event={"ID":"08ee3e3c-edfb-4820-b8fa-71387bcb490d","Type":"ContainerStarted","Data":"950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a"}
Oct 30 03:19:45.988488 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:45.988366    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podStartSLOduration=3.224724208 podStartE2EDuration="56.98834719s" totalImagesPullingTime="51.49056366s" totalInitContainerRuntime="2.273059322s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:45.975595324 +0000 UTC m=+1450.181775541" watchObservedRunningTime="2026-10-30 03:19:45.98834719 +0000 UTC m=+1450.194527411"
Oct 30 03:19:46.018730 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:46.018700    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:19:50.018947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:50.018910    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:19:50.020383 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:50.020358    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Oct 30 03:19:53.673164 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:53.673124    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:53.685971 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:53.685935    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:53.959082 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:19:53.958991    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:19:54.085070 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:19:54.085033    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:20:00.019876 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:00.019828    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Oct 30 03:20:10.020105 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:10.020051    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Oct 30 03:20:17.483022 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.482938    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"]
Oct 30 03:20:17.493277 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:20:17.493226    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2" cmd=["/bin/sleep","15"]
Oct 30 03:20:17.493459 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.493309    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:20:17.493459 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:20:17.493328    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls" podUID="ad6f1c3c-178b-4813-b19f-777a3b49532c" containerName="main" containerID="cri-o://8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2"
Oct 30 03:20:17.493459 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.493391    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls" podUID="ad6f1c3c-178b-4813-b19f-777a3b49532c" containerName="main" containerID="cri-o://8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2" gracePeriod=60
Oct 30 03:20:17.844121 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.844084    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:20:17.939066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.939020    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad6f1c3c-178b-4813-b19f-777a3b49532c-kube-api-access-d96fc\" (UniqueName: \"kubernetes.io/projected/ad6f1c3c-178b-4813-b19f-777a3b49532c-kube-api-access-d96fc\") pod \"ad6f1c3c-178b-4813-b19f-777a3b49532c\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") "
Oct 30 03:20:17.939289 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.939103    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-tmp-dir\") pod \"ad6f1c3c-178b-4813-b19f-777a3b49532c\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") "
Oct 30 03:20:17.939289 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.939204    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad6f1c3c-178b-4813-b19f-777a3b49532c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad6f1c3c-178b-4813-b19f-777a3b49532c-tls-certs\") pod \"ad6f1c3c-178b-4813-b19f-777a3b49532c\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") "
Oct 30 03:20:17.939289 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.939235    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-dshm\") pod \"ad6f1c3c-178b-4813-b19f-777a3b49532c\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") "
Oct 30 03:20:17.939465 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.939352    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-model-cache\") pod \"ad6f1c3c-178b-4813-b19f-777a3b49532c\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") "
Oct 30 03:20:17.939465 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.939390    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-home\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-home\") pod \"ad6f1c3c-178b-4813-b19f-777a3b49532c\" (UID: \"ad6f1c3c-178b-4813-b19f-777a3b49532c\") "
Oct 30 03:20:17.940045 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.940015    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-home" pod "ad6f1c3c-178b-4813-b19f-777a3b49532c" (UID: "ad6f1c3c-178b-4813-b19f-777a3b49532c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:17.940586 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.940537    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-model-cache" pod "ad6f1c3c-178b-4813-b19f-777a3b49532c" (UID: "ad6f1c3c-178b-4813-b19f-777a3b49532c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:17.940779 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.940746    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-tmp-dir" pod "ad6f1c3c-178b-4813-b19f-777a3b49532c" (UID: "ad6f1c3c-178b-4813-b19f-777a3b49532c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:17.943297 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.943253    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6f1c3c-178b-4813-b19f-777a3b49532c-kube-api-access-d96fc" pod "ad6f1c3c-178b-4813-b19f-777a3b49532c" (UID: "ad6f1c3c-178b-4813-b19f-777a3b49532c"). InnerVolumeSpecName "kube-api-access-d96fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:20:17.943418 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.943391    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-dshm" pod "ad6f1c3c-178b-4813-b19f-777a3b49532c" (UID: "ad6f1c3c-178b-4813-b19f-777a3b49532c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:20:17.943850 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:17.943821    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad6f1c3c-178b-4813-b19f-777a3b49532c-tls-certs" pod "ad6f1c3c-178b-4813-b19f-777a3b49532c" (UID: "ad6f1c3c-178b-4813-b19f-777a3b49532c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:20:18.040739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.040702    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d96fc\" (UniqueName: \"kubernetes.io/projected/ad6f1c3c-178b-4813-b19f-777a3b49532c-kube-api-access-d96fc\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:20:18.040739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.040733    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:20:18.040739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.040744    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad6f1c3c-178b-4813-b19f-777a3b49532c-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:20:18.040739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.040754    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:20:18.040739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.040761    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:20:18.040739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.040768    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad6f1c3c-178b-4813-b19f-777a3b49532c-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:20:18.072544 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.072501    4494 generic.go:356] "Generic (PLEG): container finished" podID="ad6f1c3c-178b-4813-b19f-777a3b49532c" containerID="8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2" exitCode=0
Oct 30 03:20:18.072703 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.072546    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"
Oct 30 03:20:18.072703 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.072558    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls" event={"ID":"ad6f1c3c-178b-4813-b19f-777a3b49532c","Type":"ContainerDied","Data":"8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2"}
Oct 30 03:20:18.072703 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.072617    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls" event={"ID":"ad6f1c3c-178b-4813-b19f-777a3b49532c","Type":"ContainerDied","Data":"f7d7e4fb59b28846f4fe28f2208ba7b0a2b1719d8ca9cea7041403c1ffabeffd"}
Oct 30 03:20:18.072703 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.072650    4494 scope.go:122] "RemoveContainer" containerID="8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2"
Oct 30 03:20:18.083252 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.083227    4494 scope.go:122] "RemoveContainer" containerID="8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2"
Oct 30 03:20:18.083600 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:20:18.083577    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2\": container with ID starting with 8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2 not found: ID does not exist" containerID="8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2"
Oct 30 03:20:18.083668 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.083615    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2"} err="failed to get container status \"8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2\": rpc error: code = NotFound desc = could not find container \"8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2\": container with ID starting with 8e41481bfa62823a8d28accf6019d2754a79f1eb57b7b7b2ee2544a893c7e6b2 not found: ID does not exist"
Oct 30 03:20:18.123839 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.123806    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"]
Oct 30 03:20:18.127633 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:18.127606    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-599dcb8f76rhwls"]
Oct 30 03:20:19.928479 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:19.928444    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad6f1c3c-178b-4813-b19f-777a3b49532c" path="/var/lib/kubelet/pods/ad6f1c3c-178b-4813-b19f-777a3b49532c/volumes"
Oct 30 03:20:20.019195 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:20.019142    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Oct 30 03:20:24.082083 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:20:24.082036    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:20:30.019079 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:30.019023    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Oct 30 03:20:40.019571 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:40.019521    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Oct 30 03:20:50.019183 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:20:50.019139    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Oct 30 03:20:54.084433 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:20:54.084385    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:21:00.019336 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:00.019292    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Oct 30 03:21:10.029117 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:10.029079    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:21:10.036727 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:10.036705    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:21:16.756580 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:16.756545    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"]
Oct 30 03:21:24.083165 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:21:24.083129    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:21:31.777246 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:31.777170    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main" containerID="cri-o://950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a" gracePeriod=45
Oct 30 03:21:32.030252 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.030161    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": dial tcp 10.134.0.36:8000: connect: connection refused"
Oct 30 03:21:32.634262 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.634233    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:21:32.764101 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764010    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ee3e3c-edfb-4820-b8fa-71387bcb490d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08ee3e3c-edfb-4820-b8fa-71387bcb490d-cabundle-cert\") pod \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") "
Oct 30 03:21:32.764101 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764055    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-model-cache\") pod \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") "
Oct 30 03:21:32.764101 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764077    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kube-api-access-xbv8n\" (UniqueName: \"kubernetes.io/projected/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kube-api-access-xbv8n\") pod \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") "
Oct 30 03:21:32.764393 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764127    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kserve-provision-location\") pod \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") "
Oct 30 03:21:32.764393 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764167    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tmp-dir\") pod \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") "
Oct 30 03:21:32.764393 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764184    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tls-certs\" (UniqueName: \"kubernetes.io/secret/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tls-certs\") pod \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") "
Oct 30 03:21:32.764393 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764203    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-home\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-home\") pod \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") "
Oct 30 03:21:32.764393 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764225    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-dshm\") pod \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\" (UID: \"08ee3e3c-edfb-4820-b8fa-71387bcb490d\") "
Oct 30 03:21:32.764627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764361    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-model-cache" pod "08ee3e3c-edfb-4820-b8fa-71387bcb490d" (UID: "08ee3e3c-edfb-4820-b8fa-71387bcb490d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:32.764627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764463    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ee3e3c-edfb-4820-b8fa-71387bcb490d-cabundle-cert" pod "08ee3e3c-edfb-4820-b8fa-71387bcb490d" (UID: "08ee3e3c-edfb-4820-b8fa-71387bcb490d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:21:32.764745 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764721    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08ee3e3c-edfb-4820-b8fa-71387bcb490d-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:21:32.764799 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764749    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:21:32.764840 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.764799    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-home" pod "08ee3e3c-edfb-4820-b8fa-71387bcb490d" (UID: "08ee3e3c-edfb-4820-b8fa-71387bcb490d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:32.766804 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.766770    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tls-certs" pod "08ee3e3c-edfb-4820-b8fa-71387bcb490d" (UID: "08ee3e3c-edfb-4820-b8fa-71387bcb490d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:21:32.766941 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.766882    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kube-api-access-xbv8n" pod "08ee3e3c-edfb-4820-b8fa-71387bcb490d" (UID: "08ee3e3c-edfb-4820-b8fa-71387bcb490d"). InnerVolumeSpecName "kube-api-access-xbv8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:21:32.767012 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.766951    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-dshm" pod "08ee3e3c-edfb-4820-b8fa-71387bcb490d" (UID: "08ee3e3c-edfb-4820-b8fa-71387bcb490d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:32.775930 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.775883    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tmp-dir" pod "08ee3e3c-edfb-4820-b8fa-71387bcb490d" (UID: "08ee3e3c-edfb-4820-b8fa-71387bcb490d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:32.786641 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.786609    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kserve-provision-location" pod "08ee3e3c-edfb-4820-b8fa-71387bcb490d" (UID: "08ee3e3c-edfb-4820-b8fa-71387bcb490d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:21:32.865950 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.865884    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbv8n\" (UniqueName: \"kubernetes.io/projected/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kube-api-access-xbv8n\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:21:32.865950 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.865947    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:21:32.866139 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.865980    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:21:32.866139 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.865991    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ee3e3c-edfb-4820-b8fa-71387bcb490d-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:21:32.866139 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.866001    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:21:32.866139 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:32.866009    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ee3e3c-edfb-4820-b8fa-71387bcb490d-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:21:33.433516 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.433479    4494 generic.go:356] "Generic (PLEG): container finished" podID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerID="950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a" exitCode=0
Oct 30 03:21:33.433706 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.433527    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"
Oct 30 03:21:33.433706 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.433561    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" event={"ID":"08ee3e3c-edfb-4820-b8fa-71387bcb490d","Type":"ContainerDied","Data":"950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a"}
Oct 30 03:21:33.433706 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.433609    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh" event={"ID":"08ee3e3c-edfb-4820-b8fa-71387bcb490d","Type":"ContainerDied","Data":"1356fff69f0e2a2a5c3f040654f48dd1ee48832603358278d70bf6cba0fd901d"}
Oct 30 03:21:33.433706 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.433633    4494 scope.go:122] "RemoveContainer" containerID="950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a"
Oct 30 03:21:33.443183 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.443168    4494 scope.go:122] "RemoveContainer" containerID="e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4"
Oct 30 03:21:33.451384 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.451365    4494 scope.go:122] "RemoveContainer" containerID="950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a"
Oct 30 03:21:33.451663 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:21:33.451643    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a\": container with ID starting with 950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a not found: ID does not exist" containerID="950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a"
Oct 30 03:21:33.451707 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.451674    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a"} err="failed to get container status \"950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a\": rpc error: code = NotFound desc = could not find container \"950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a\": container with ID starting with 950aa665511519a12b71ead52093a64e0cfc2fead57a6dc307cdaf3baff0432a not found: ID does not exist"
Oct 30 03:21:33.451707 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.451693    4494 scope.go:122] "RemoveContainer" containerID="e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4"
Oct 30 03:21:33.451958 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:21:33.451940    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4\": container with ID starting with e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4 not found: ID does not exist" containerID="e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4"
Oct 30 03:21:33.452006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.451965    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4"} err="failed to get container status \"e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4\": rpc error: code = NotFound desc = could not find container \"e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4\": container with ID starting with e9faf5f7a8b8f7e93f70da6513da4b73970c39603f0aab885409bd53f0c410c4 not found: ID does not exist"
Oct 30 03:21:33.499157 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.499122    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"]
Oct 30 03:21:33.511702 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.511654    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-6b7f9c57b-xs2kh"]
Oct 30 03:21:33.927308 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:21:33.927275    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" path="/var/lib/kubelet/pods/08ee3e3c-edfb-4820-b8fa-71387bcb490d/volumes"
Oct 30 03:21:54.082251 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:21:54.082165    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:22:11.027423 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.027389    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"]
Oct 30 03:22:11.027957 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.027938    4494 cpu_manager.go:414] "containerMap: removing container" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main"
Oct 30 03:22:11.028009 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.027958    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main"
Oct 30 03:22:11.028009 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.027989    4494 cpu_manager.go:414] "containerMap: removing container" podUID="ad6f1c3c-178b-4813-b19f-777a3b49532c" containerName="main"
Oct 30 03:22:11.028009 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.027997    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad6f1c3c-178b-4813-b19f-777a3b49532c" containerName="main"
Oct 30 03:22:11.028113 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.028017    4494 cpu_manager.go:414] "containerMap: removing container" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="storage-initializer"
Oct 30 03:22:11.028113 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.028026    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="storage-initializer"
Oct 30 03:22:11.028113 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.028100    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad6f1c3c-178b-4813-b19f-777a3b49532c" containerName="main"
Oct 30 03:22:11.028113 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.028112    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ee3e3c-edfb-4820-b8fa-71387bcb490d" containerName="main"
Oct 30 03:22:11.031292 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.031272    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.062663 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.062634    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Oct 30 03:22:11.062663 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.062659    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-6lnhh\""
Oct 30 03:22:11.062913 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.062668    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Oct 30 03:22:11.062913 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.062668    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Oct 30 03:22:11.064317 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.064293    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"]
Oct 30 03:22:11.111140 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.111102    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d393e3a-43e9-4efb-bbe6-786a97375d79-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.111332 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.111247    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.111332 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.111290    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.111332 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.111306    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.111436 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.111334    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.111469 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.111447    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smc6k\" (UniqueName: \"kubernetes.io/projected/0d393e3a-43e9-4efb-bbe6-786a97375d79-kube-api-access-smc6k\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.212227 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.212188    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.212227 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.212231    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.212501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.212248    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.212501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.212271    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.212501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.212300    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smc6k\" (UniqueName: \"kubernetes.io/projected/0d393e3a-43e9-4efb-bbe6-786a97375d79-kube-api-access-smc6k\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.212501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.212318    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d393e3a-43e9-4efb-bbe6-786a97375d79-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.212709 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.212672    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.212752 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.212714    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.212785 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.212762    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.214681 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.214657    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.214843 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.214825    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d393e3a-43e9-4efb-bbe6-786a97375d79-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.235700 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.235673    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smc6k\" (UniqueName: \"kubernetes.io/projected/0d393e3a-43e9-4efb-bbe6-786a97375d79-kube-api-access-smc6k\") pod \"fc-smoke-test-kserve-7d8f9d9664-rzbt2\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.344434 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.344297    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:11.639780 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:11.639755    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"]
Oct 30 03:22:11.639976 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:22:11.639953    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d393e3a_43e9_4efb_bbe6_786a97375d79.slice/crio-1956fa13fedaf01447a06f61448c10b2ea2277f1f9775dfbac793b07affa68a8 WatchSource:0}: Error finding container 1956fa13fedaf01447a06f61448c10b2ea2277f1f9775dfbac793b07affa68a8: Status 404 returned error can't find the container with id 1956fa13fedaf01447a06f61448c10b2ea2277f1f9775dfbac793b07affa68a8
Oct 30 03:22:12.611051 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:12.611013    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2" event={"ID":"0d393e3a-43e9-4efb-bbe6-786a97375d79","Type":"ContainerStarted","Data":"4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97"}
Oct 30 03:22:12.611526 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:12.611066    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2" event={"ID":"0d393e3a-43e9-4efb-bbe6-786a97375d79","Type":"ContainerStarted","Data":"1956fa13fedaf01447a06f61448c10b2ea2277f1f9775dfbac793b07affa68a8"}
Oct 30 03:22:13.344788 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:13.344738    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:21.344788 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:21.344751    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:21.357240 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:21.357216    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:21.399146 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:21.399098    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2" podStartSLOduration=11.399072627 podStartE2EDuration="11.399072627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:12.6745232 +0000 UTC m=+1596.880703418" watchObservedRunningTime="2026-10-30 03:22:21.399072627 +0000 UTC m=+1605.605252845"
Oct 30 03:22:21.661973 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:21.661877    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:24.081033 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:22:24.080996    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:22:51.776580 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:51.776545    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"]
Oct 30 03:22:51.786673 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:22:51.786624    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97" cmd=["/bin/sleep","15"]
Oct 30 03:22:51.786846 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:51.786693    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:22:51.786846 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:22:51.786721    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2" podUID="0d393e3a-43e9-4efb-bbe6-786a97375d79" containerName="main" containerID="cri-o://4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97"
Oct 30 03:22:51.786846 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:51.786776    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2" podUID="0d393e3a-43e9-4efb-bbe6-786a97375d79" containerName="main" containerID="cri-o://4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97" gracePeriod=60
Oct 30 03:22:52.133641 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.133612    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:52.208379 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208341    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-home\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-home\") pod \"0d393e3a-43e9-4efb-bbe6-786a97375d79\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") "
Oct 30 03:22:52.208572 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208395    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d393e3a-43e9-4efb-bbe6-786a97375d79-kube-api-access-smc6k\" (UniqueName: \"kubernetes.io/projected/0d393e3a-43e9-4efb-bbe6-786a97375d79-kube-api-access-smc6k\") pod \"0d393e3a-43e9-4efb-bbe6-786a97375d79\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") "
Oct 30 03:22:52.208572 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208416    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-model-cache\") pod \"0d393e3a-43e9-4efb-bbe6-786a97375d79\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") "
Oct 30 03:22:52.208572 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208461    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d393e3a-43e9-4efb-bbe6-786a97375d79-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d393e3a-43e9-4efb-bbe6-786a97375d79-tls-certs\") pod \"0d393e3a-43e9-4efb-bbe6-786a97375d79\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") "
Oct 30 03:22:52.208572 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208542    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-tmp-dir\") pod \"0d393e3a-43e9-4efb-bbe6-786a97375d79\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") "
Oct 30 03:22:52.208795 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208606    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-dshm\") pod \"0d393e3a-43e9-4efb-bbe6-786a97375d79\" (UID: \"0d393e3a-43e9-4efb-bbe6-786a97375d79\") "
Oct 30 03:22:52.208795 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208614    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-home" pod "0d393e3a-43e9-4efb-bbe6-786a97375d79" (UID: "0d393e3a-43e9-4efb-bbe6-786a97375d79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:52.208795 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208717    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-model-cache" pod "0d393e3a-43e9-4efb-bbe6-786a97375d79" (UID: "0d393e3a-43e9-4efb-bbe6-786a97375d79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:52.208994 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208975    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:22:52.209049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208972    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-tmp-dir" pod "0d393e3a-43e9-4efb-bbe6-786a97375d79" (UID: "0d393e3a-43e9-4efb-bbe6-786a97375d79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:52.209049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.208999    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:22:52.210959 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.210927    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-dshm" pod "0d393e3a-43e9-4efb-bbe6-786a97375d79" (UID: "0d393e3a-43e9-4efb-bbe6-786a97375d79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:22:52.211103 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.210961    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d393e3a-43e9-4efb-bbe6-786a97375d79-tls-certs" pod "0d393e3a-43e9-4efb-bbe6-786a97375d79" (UID: "0d393e3a-43e9-4efb-bbe6-786a97375d79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:22:52.211103 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.211062    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d393e3a-43e9-4efb-bbe6-786a97375d79-kube-api-access-smc6k" pod "0d393e3a-43e9-4efb-bbe6-786a97375d79" (UID: "0d393e3a-43e9-4efb-bbe6-786a97375d79"). InnerVolumeSpecName "kube-api-access-smc6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:22:52.310501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.310407    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:22:52.310501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.310438    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d393e3a-43e9-4efb-bbe6-786a97375d79-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:22:52.310501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.310448    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smc6k\" (UniqueName: \"kubernetes.io/projected/0d393e3a-43e9-4efb-bbe6-786a97375d79-kube-api-access-smc6k\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:22:52.310501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.310456    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d393e3a-43e9-4efb-bbe6-786a97375d79-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:22:52.792317 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.792282    4494 generic.go:356] "Generic (PLEG): container finished" podID="0d393e3a-43e9-4efb-bbe6-786a97375d79" containerID="4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97" exitCode=0
Oct 30 03:22:52.792796 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.792334    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"
Oct 30 03:22:52.792796 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.792369    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2" event={"ID":"0d393e3a-43e9-4efb-bbe6-786a97375d79","Type":"ContainerDied","Data":"4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97"}
Oct 30 03:22:52.792796 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.792423    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2" event={"ID":"0d393e3a-43e9-4efb-bbe6-786a97375d79","Type":"ContainerDied","Data":"1956fa13fedaf01447a06f61448c10b2ea2277f1f9775dfbac793b07affa68a8"}
Oct 30 03:22:52.792796 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.792445    4494 scope.go:122] "RemoveContainer" containerID="4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97"
Oct 30 03:22:52.801624 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.801610    4494 scope.go:122] "RemoveContainer" containerID="4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97"
Oct 30 03:22:52.801918 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:22:52.801877    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97\": container with ID starting with 4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97 not found: ID does not exist" containerID="4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97"
Oct 30 03:22:52.802016 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.801925    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97"} err="failed to get container status \"4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97\": rpc error: code = NotFound desc = could not find container \"4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97\": container with ID starting with 4f5cc8625a442d4acc327b7c128edc9bf0443576f041400d81adadbe5883ca97 not found: ID does not exist"
Oct 30 03:22:52.843668 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.843630    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"]
Oct 30 03:22:52.855836 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:52.855803    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-rzbt2"]
Oct 30 03:22:53.926955 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:22:53.926925    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d393e3a-43e9-4efb-bbe6-786a97375d79" path="/var/lib/kubelet/pods/0d393e3a-43e9-4efb-bbe6-786a97375d79/volumes"
Oct 30 03:22:54.085459 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:22:54.085419    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:23:24.081845 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:23:24.081808    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:23:45.948921 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:45.948870    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"]
Oct 30 03:23:45.949458 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:45.949403    4494 cpu_manager.go:414] "containerMap: removing container" podUID="0d393e3a-43e9-4efb-bbe6-786a97375d79" containerName="main"
Oct 30 03:23:45.949458 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:45.949419    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d393e3a-43e9-4efb-bbe6-786a97375d79" containerName="main"
Oct 30 03:23:45.949574 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:45.949506    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d393e3a-43e9-4efb-bbe6-786a97375d79" containerName="main"
Oct 30 03:23:45.953859 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:45.953835    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:45.983440 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:45.983398    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Oct 30 03:23:45.983620 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:45.983398    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Oct 30 03:23:45.983620 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:45.983459    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Oct 30 03:23:45.983620 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:45.983547    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-k8lxx\""
Oct 30 03:23:45.985032 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:45.985010    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"]
Oct 30 03:23:46.103588 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.103525    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-dshm\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.103588 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.103582    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tmp-dir\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.104008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.103665    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tls-certs\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.104008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.103697    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-home\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.104008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.103765    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-model-cache\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.104008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.103832    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2nw\" (UniqueName: \"kubernetes.io/projected/44b8a200-8aa9-4eb3-853e-bccf1347aacf-kube-api-access-rd2nw\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.204993 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.204842    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-model-cache\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.204993 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.204938    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2nw\" (UniqueName: \"kubernetes.io/projected/44b8a200-8aa9-4eb3-853e-bccf1347aacf-kube-api-access-rd2nw\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.204993 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.204989    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-dshm\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.205280 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.205009    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tmp-dir\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.205280 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.205148    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tls-certs\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.205280 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.205190    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-home\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.205443 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.205330    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-model-cache\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.205443 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.205383    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tmp-dir\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.205557 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.205475    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-home\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.207541 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.207516    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-dshm\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.207736 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.207717    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tls-certs\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.228752 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.228724    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2nw\" (UniqueName: \"kubernetes.io/projected/44b8a200-8aa9-4eb3-853e-bccf1347aacf-kube-api-access-rd2nw\") pod \"fc-concurrency-test-kserve-5485c7876d-4dzwd\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.269151 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.269118    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:46.564130 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:46.564104    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"]
Oct 30 03:23:46.565189 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:23:46.565157    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b8a200_8aa9_4eb3_853e_bccf1347aacf.slice/crio-7b823b88fd2da4881f45aeef9c7e5f6c815b6e826c880090cf30645f39692603 WatchSource:0}: Error finding container 7b823b88fd2da4881f45aeef9c7e5f6c815b6e826c880090cf30645f39692603: Status 404 returned error can't find the container with id 7b823b88fd2da4881f45aeef9c7e5f6c815b6e826c880090cf30645f39692603
Oct 30 03:23:47.037222 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:47.037181    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd" event={"ID":"44b8a200-8aa9-4eb3-853e-bccf1347aacf","Type":"ContainerStarted","Data":"e16404bb276fe8707b0a0ca3364b1c24c637ee643596f9074d698afc0b2fb99e"}
Oct 30 03:23:47.037222 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:47.037228    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd" event={"ID":"44b8a200-8aa9-4eb3-853e-bccf1347aacf","Type":"ContainerStarted","Data":"7b823b88fd2da4881f45aeef9c7e5f6c815b6e826c880090cf30645f39692603"}
Oct 30 03:23:47.082077 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:47.082029    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd" podStartSLOduration=2.08201024 podStartE2EDuration="2.08201024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:47.081297116 +0000 UTC m=+1691.287477334" watchObservedRunningTime="2026-10-30 03:23:47.08201024 +0000 UTC m=+1691.288190458"
Oct 30 03:23:47.270050 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:47.270016    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:54.082339 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:23:54.082293    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:23:56.269380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:56.269336    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:56.282266 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:56.282239    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:57.094083 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:57.094057    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:23:59.483383 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:59.483340    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"]
Oct 30 03:23:59.483868 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:59.483603    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerName="istio-proxy" containerID="cri-o://5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039" gracePeriod=30
Oct 30 03:23:59.496028 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:59.495992    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"]
Oct 30 03:23:59.496296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:23:59.496254    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerName="istio-proxy" containerID="cri-o://caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add" gracePeriod=30
Oct 30 03:24:04.742860 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.742834    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:24:04.767743 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.767711    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-socket\") pod \"41414588-b1f1-4bbe-bc40-552dd3ad266c\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") "
Oct 30 03:24:04.767893 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.767764    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-data\") pod \"41414588-b1f1-4bbe-bc40-552dd3ad266c\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") "
Oct 30 03:24:04.767893 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.767810    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-kube-api-access-f66tf\" (UniqueName: \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-kube-api-access-f66tf\") pod \"41414588-b1f1-4bbe-bc40-552dd3ad266c\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") "
Oct 30 03:24:04.767893 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.767828    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-certs\") pod \"41414588-b1f1-4bbe-bc40-552dd3ad266c\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") "
Oct 30 03:24:04.767893 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.767854    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-podinfo\") pod \"41414588-b1f1-4bbe-bc40-552dd3ad266c\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") "
Oct 30 03:24:04.768130 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.767890    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-token\" (UniqueName: \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-token\") pod \"41414588-b1f1-4bbe-bc40-552dd3ad266c\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") "
Oct 30 03:24:04.768130 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.767959    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-credential-socket\") pod \"41414588-b1f1-4bbe-bc40-552dd3ad266c\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") "
Oct 30 03:24:04.768130 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768011    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41414588-b1f1-4bbe-bc40-552dd3ad266c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41414588-b1f1-4bbe-bc40-552dd3ad266c-istiod-ca-cert\") pod \"41414588-b1f1-4bbe-bc40-552dd3ad266c\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") "
Oct 30 03:24:04.768130 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768020    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-socket" pod "41414588-b1f1-4bbe-bc40-552dd3ad266c" (UID: "41414588-b1f1-4bbe-bc40-552dd3ad266c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:04.768130 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768052    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-envoy\") pod \"41414588-b1f1-4bbe-bc40-552dd3ad266c\" (UID: \"41414588-b1f1-4bbe-bc40-552dd3ad266c\") "
Oct 30 03:24:04.768373 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768164    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-data" pod "41414588-b1f1-4bbe-bc40-552dd3ad266c" (UID: "41414588-b1f1-4bbe-bc40-552dd3ad266c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:04.768373 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768181    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-certs" pod "41414588-b1f1-4bbe-bc40-552dd3ad266c" (UID: "41414588-b1f1-4bbe-bc40-552dd3ad266c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:04.768373 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768339    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-credential-socket" pod "41414588-b1f1-4bbe-bc40-552dd3ad266c" (UID: "41414588-b1f1-4bbe-bc40-552dd3ad266c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:04.768475 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768392    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.768475 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768411    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.768475 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768425    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.768475 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768458    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41414588-b1f1-4bbe-bc40-552dd3ad266c-istiod-ca-cert" pod "41414588-b1f1-4bbe-bc40-552dd3ad266c" (UID: "41414588-b1f1-4bbe-bc40-552dd3ad266c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:04.768830 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.768810    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:24:04.770427 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.770404    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-kube-api-access-f66tf" pod "41414588-b1f1-4bbe-bc40-552dd3ad266c" (UID: "41414588-b1f1-4bbe-bc40-552dd3ad266c"). InnerVolumeSpecName "kube-api-access-f66tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:04.770427 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.770411    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-podinfo" pod "41414588-b1f1-4bbe-bc40-552dd3ad266c" (UID: "41414588-b1f1-4bbe-bc40-552dd3ad266c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:04.770587 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.770505    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-envoy" pod "41414588-b1f1-4bbe-bc40-552dd3ad266c" (UID: "41414588-b1f1-4bbe-bc40-552dd3ad266c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:04.770587 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.770570    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-token" pod "41414588-b1f1-4bbe-bc40-552dd3ad266c" (UID: "41414588-b1f1-4bbe-bc40-552dd3ad266c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:04.868982 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.868941    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-data\") pod \"2dda10f8-5171-473e-8e69-4c104201f3ee\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") "
Oct 30 03:24:04.869180 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869003    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-kube-api-access-8fjnq\" (UniqueName: \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-kube-api-access-8fjnq\") pod \"2dda10f8-5171-473e-8e69-4c104201f3ee\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") "
Oct 30 03:24:04.869180 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869039    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dda10f8-5171-473e-8e69-4c104201f3ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dda10f8-5171-473e-8e69-4c104201f3ee-istiod-ca-cert\") pod \"2dda10f8-5171-473e-8e69-4c104201f3ee\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") "
Oct 30 03:24:04.869180 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869085    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-envoy\") pod \"2dda10f8-5171-473e-8e69-4c104201f3ee\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") "
Oct 30 03:24:04.869180 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869134    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-podinfo\") pod \"2dda10f8-5171-473e-8e69-4c104201f3ee\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") "
Oct 30 03:24:04.869395 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869183    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-token\" (UniqueName: \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-token\") pod \"2dda10f8-5171-473e-8e69-4c104201f3ee\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") "
Oct 30 03:24:04.869395 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869229    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-credential-socket\") pod \"2dda10f8-5171-473e-8e69-4c104201f3ee\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") "
Oct 30 03:24:04.869395 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869257    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-certs\") pod \"2dda10f8-5171-473e-8e69-4c104201f3ee\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") "
Oct 30 03:24:04.869395 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869316    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-socket\") pod \"2dda10f8-5171-473e-8e69-4c104201f3ee\" (UID: \"2dda10f8-5171-473e-8e69-4c104201f3ee\") "
Oct 30 03:24:04.869395 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869326    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-data" pod "2dda10f8-5171-473e-8e69-4c104201f3ee" (UID: "2dda10f8-5171-473e-8e69-4c104201f3ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:04.869694 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869513    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dda10f8-5171-473e-8e69-4c104201f3ee-istiod-ca-cert" pod "2dda10f8-5171-473e-8e69-4c104201f3ee" (UID: "2dda10f8-5171-473e-8e69-4c104201f3ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:24:04.869694 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869599    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-certs" pod "2dda10f8-5171-473e-8e69-4c104201f3ee" (UID: "2dda10f8-5171-473e-8e69-4c104201f3ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:04.869694 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869633    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.869694 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869667    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66tf\" (UniqueName: \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-kube-api-access-f66tf\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.869694 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869685    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.869973 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869701    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.869973 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869716    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.869973 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869717    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-credential-socket" pod "2dda10f8-5171-473e-8e69-4c104201f3ee" (UID: "2dda10f8-5171-473e-8e69-4c104201f3ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:04.869973 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869729    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41414588-b1f1-4bbe-bc40-552dd3ad266c-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.869973 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869742    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41414588-b1f1-4bbe-bc40-552dd3ad266c-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.869973 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.869834    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-socket" pod "2dda10f8-5171-473e-8e69-4c104201f3ee" (UID: "2dda10f8-5171-473e-8e69-4c104201f3ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:04.871945 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.871889    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-token" pod "2dda10f8-5171-473e-8e69-4c104201f3ee" (UID: "2dda10f8-5171-473e-8e69-4c104201f3ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:04.872060 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.871998    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-kube-api-access-8fjnq" pod "2dda10f8-5171-473e-8e69-4c104201f3ee" (UID: "2dda10f8-5171-473e-8e69-4c104201f3ee"). InnerVolumeSpecName "kube-api-access-8fjnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:04.872060 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.872007    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-envoy" pod "2dda10f8-5171-473e-8e69-4c104201f3ee" (UID: "2dda10f8-5171-473e-8e69-4c104201f3ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:04.872356 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.872333    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-podinfo" pod "2dda10f8-5171-473e-8e69-4c104201f3ee" (UID: "2dda10f8-5171-473e-8e69-4c104201f3ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:24:04.971163 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.971128    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.971163 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.971162    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.971374 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.971178    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.971374 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.971193    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.971374 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.971206    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fjnq\" (UniqueName: \"kubernetes.io/projected/2dda10f8-5171-473e-8e69-4c104201f3ee-kube-api-access-8fjnq\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.971374 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.971222    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dda10f8-5171-473e-8e69-4c104201f3ee-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.971374 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.971235    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:04.971374 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:04.971250    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dda10f8-5171-473e-8e69-4c104201f3ee-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:05.118305 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.118266    4494 generic.go:356] "Generic (PLEG): container finished" podID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerID="caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add" exitCode=0
Oct 30 03:24:05.118493 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.118324    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"
Oct 30 03:24:05.118493 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.118357    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" event={"ID":"2dda10f8-5171-473e-8e69-4c104201f3ee","Type":"ContainerDied","Data":"caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add"}
Oct 30 03:24:05.118493 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.118417    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb" event={"ID":"2dda10f8-5171-473e-8e69-4c104201f3ee","Type":"ContainerDied","Data":"76f0dd124165ff8def03c0c9cbeebcaf915482326a0bb64ae8153c689a103b4b"}
Oct 30 03:24:05.118493 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.118440    4494 scope.go:122] "RemoveContainer" containerID="caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add"
Oct 30 03:24:05.120016 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.119989    4494 generic.go:356] "Generic (PLEG): container finished" podID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerID="5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039" exitCode=0
Oct 30 03:24:05.120143 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.120044    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"
Oct 30 03:24:05.120143 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.120073    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" event={"ID":"41414588-b1f1-4bbe-bc40-552dd3ad266c","Type":"ContainerDied","Data":"5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039"}
Oct 30 03:24:05.120143 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.120134    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr" event={"ID":"41414588-b1f1-4bbe-bc40-552dd3ad266c","Type":"ContainerDied","Data":"8fe91018f1e13e7b5d4745b58ad7a7a869a0dd94dbbe7ce609beb45131b9793c"}
Oct 30 03:24:05.127488 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.127469    4494 scope.go:122] "RemoveContainer" containerID="caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add"
Oct 30 03:24:05.127798 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:24:05.127778    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add\": container with ID starting with caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add not found: ID does not exist" containerID="caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add"
Oct 30 03:24:05.128039 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.127808    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add"} err="failed to get container status \"caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add\": rpc error: code = NotFound desc = could not find container \"caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add\": container with ID starting with caa455d623b2b125f7a9fe21ff456252675341abee537cb5618babcec4560add not found: ID does not exist"
Oct 30 03:24:05.128039 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.127828    4494 scope.go:122] "RemoveContainer" containerID="5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039"
Oct 30 03:24:05.137651 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.137616    4494 scope.go:122] "RemoveContainer" containerID="5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039"
Oct 30 03:24:05.137974 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:24:05.137937    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039\": container with ID starting with 5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039 not found: ID does not exist" containerID="5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039"
Oct 30 03:24:05.138054 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.137969    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039"} err="failed to get container status \"5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039\": rpc error: code = NotFound desc = could not find container \"5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039\": container with ID starting with 5354dcaee822c0a9912272c0b20001bb936d333a327562d349fd71944da92039 not found: ID does not exist"
Oct 30 03:24:05.170092 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.170054    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"]
Oct 30 03:24:05.182341 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.182311    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-j58vr"]
Oct 30 03:24:05.216685 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.216649    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"]
Oct 30 03:24:05.220864 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.220835    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-v9lwb"]
Oct 30 03:24:05.926855 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.926817    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" path="/var/lib/kubelet/pods/2dda10f8-5171-473e-8e69-4c104201f3ee/volumes"
Oct 30 03:24:05.927283 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:05.927254    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" path="/var/lib/kubelet/pods/41414588-b1f1-4bbe-bc40-552dd3ad266c/volumes"
Oct 30 03:24:24.079883 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:24:24.079844    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:24:25.882754 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:25.882720    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"]
Oct 30 03:24:25.892190 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:24:25.892154    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="e16404bb276fe8707b0a0ca3364b1c24c637ee643596f9074d698afc0b2fb99e" cmd=["/bin/sleep","15"]
Oct 30 03:24:25.892342 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:25.892210    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:24:25.892342 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:24:25.892223    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd" podUID="44b8a200-8aa9-4eb3-853e-bccf1347aacf" containerName="main" containerID="cri-o://e16404bb276fe8707b0a0ca3364b1c24c637ee643596f9074d698afc0b2fb99e"
Oct 30 03:24:25.892342 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:25.892258    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd" podUID="44b8a200-8aa9-4eb3-853e-bccf1347aacf" containerName="main" containerID="cri-o://e16404bb276fe8707b0a0ca3364b1c24c637ee643596f9074d698afc0b2fb99e" gracePeriod=60
Oct 30 03:24:26.083291 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.083258    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd" podUID="44b8a200-8aa9-4eb3-853e-bccf1347aacf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Oct 30 03:24:26.213953 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.213861    4494 generic.go:356] "Generic (PLEG): container finished" podID="44b8a200-8aa9-4eb3-853e-bccf1347aacf" containerID="e16404bb276fe8707b0a0ca3364b1c24c637ee643596f9074d698afc0b2fb99e" exitCode=0
Oct 30 03:24:26.213953 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.213927    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd" event={"ID":"44b8a200-8aa9-4eb3-853e-bccf1347aacf","Type":"ContainerDied","Data":"e16404bb276fe8707b0a0ca3364b1c24c637ee643596f9074d698afc0b2fb99e"}
Oct 30 03:24:26.633336 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.633314    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:24:26.756223 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.756131    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b8a200-8aa9-4eb3-853e-bccf1347aacf-kube-api-access-rd2nw\" (UniqueName: \"kubernetes.io/projected/44b8a200-8aa9-4eb3-853e-bccf1347aacf-kube-api-access-rd2nw\") pod \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") "
Oct 30 03:24:26.756223 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.756189    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-dshm\") pod \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") "
Oct 30 03:24:26.756223 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.756222    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-home\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-home\") pod \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") "
Oct 30 03:24:26.756494 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.756282    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tmp-dir\") pod \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") "
Oct 30 03:24:26.756494 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.756342    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-model-cache\") pod \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") "
Oct 30 03:24:26.756494 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.756388    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tls-certs\" (UniqueName: \"kubernetes.io/secret/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tls-certs\") pod \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\" (UID: \"44b8a200-8aa9-4eb3-853e-bccf1347aacf\") "
Oct 30 03:24:26.756949 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.756885    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-home" pod "44b8a200-8aa9-4eb3-853e-bccf1347aacf" (UID: "44b8a200-8aa9-4eb3-853e-bccf1347aacf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:26.757058 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.756955    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tmp-dir" pod "44b8a200-8aa9-4eb3-853e-bccf1347aacf" (UID: "44b8a200-8aa9-4eb3-853e-bccf1347aacf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:26.757058 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.756966    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-model-cache" pod "44b8a200-8aa9-4eb3-853e-bccf1347aacf" (UID: "44b8a200-8aa9-4eb3-853e-bccf1347aacf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:26.758947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.758920    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tls-certs" pod "44b8a200-8aa9-4eb3-853e-bccf1347aacf" (UID: "44b8a200-8aa9-4eb3-853e-bccf1347aacf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:24:26.759431 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.759401    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-dshm" pod "44b8a200-8aa9-4eb3-853e-bccf1347aacf" (UID: "44b8a200-8aa9-4eb3-853e-bccf1347aacf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:24:26.759431 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.759418    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b8a200-8aa9-4eb3-853e-bccf1347aacf-kube-api-access-rd2nw" pod "44b8a200-8aa9-4eb3-853e-bccf1347aacf" (UID: "44b8a200-8aa9-4eb3-853e-bccf1347aacf"). InnerVolumeSpecName "kube-api-access-rd2nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:24:26.857207 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.857173    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:26.857207 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.857203    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:26.857416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.857233    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:26.857416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.857243    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44b8a200-8aa9-4eb3-853e-bccf1347aacf-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:26.857416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.857251    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44b8a200-8aa9-4eb3-853e-bccf1347aacf-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:26.857416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:26.857259    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2nw\" (UniqueName: \"kubernetes.io/projected/44b8a200-8aa9-4eb3-853e-bccf1347aacf-kube-api-access-rd2nw\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:24:27.219319 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:27.219292    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"
Oct 30 03:24:27.219688 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:27.219317    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd" event={"ID":"44b8a200-8aa9-4eb3-853e-bccf1347aacf","Type":"ContainerDied","Data":"7b823b88fd2da4881f45aeef9c7e5f6c815b6e826c880090cf30645f39692603"}
Oct 30 03:24:27.219688 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:27.219381    4494 scope.go:122] "RemoveContainer" containerID="e16404bb276fe8707b0a0ca3364b1c24c637ee643596f9074d698afc0b2fb99e"
Oct 30 03:24:27.270136 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:27.270112    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"]
Oct 30 03:24:27.282235 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:27.282213    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-4dzwd"]
Oct 30 03:24:27.926416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:24:27.926386    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b8a200-8aa9-4eb3-853e-bccf1347aacf" path="/var/lib/kubelet/pods/44b8a200-8aa9-4eb3-853e-bccf1347aacf/volumes"
Oct 30 03:24:54.080771 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:24:54.080689    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:25:16.464813 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.464780    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"]
Oct 30 03:25:16.465589 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.465568    4494 cpu_manager.go:414] "containerMap: removing container" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerName="istio-proxy"
Oct 30 03:25:16.465758 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.465743    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerName="istio-proxy"
Oct 30 03:25:16.465857 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.465837    4494 cpu_manager.go:414] "containerMap: removing container" podUID="44b8a200-8aa9-4eb3-853e-bccf1347aacf" containerName="main"
Oct 30 03:25:16.465857 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.465854    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b8a200-8aa9-4eb3-853e-bccf1347aacf" containerName="main"
Oct 30 03:25:16.466068 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.465867    4494 cpu_manager.go:414] "containerMap: removing container" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerName="istio-proxy"
Oct 30 03:25:16.466068 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.465873    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerName="istio-proxy"
Oct 30 03:25:16.466068 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.466026    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="41414588-b1f1-4bbe-bc40-552dd3ad266c" containerName="istio-proxy"
Oct 30 03:25:16.466068 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.466036    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b8a200-8aa9-4eb3-853e-bccf1347aacf" containerName="main"
Oct 30 03:25:16.466068 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.466042    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dda10f8-5171-473e-8e69-4c104201f3ee" containerName="istio-proxy"
Oct 30 03:25:16.469339 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.469318    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.486971 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.486944    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-xps78\""
Oct 30 03:25:16.487115 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.487025    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Oct 30 03:25:16.498466 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.498441    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Oct 30 03:25:16.498466 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.498443    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Oct 30 03:25:16.500110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.500090    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"]
Oct 30 03:25:16.547294 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.547254    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-certs\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.547294 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.547291    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-socket\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.547535 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.547320    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-data\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.547535 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.547432    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.547535 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.547454    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-podinfo\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.547535 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.547484    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-credential-socket\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.547535 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.547499    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-token\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.547766 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.547575    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjs5\" (UniqueName: \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-kube-api-access-lkjs5\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.547766 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.547617    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-envoy\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.648755 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.648718    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-certs\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.648755 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.648758    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-socket\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649046 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.648784    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-data\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649046 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.648819    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649046 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.648843    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-podinfo\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649046 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.648872    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-credential-socket\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649046 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.648887    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-token\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649046 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.648944    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjs5\" (UniqueName: \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-kube-api-access-lkjs5\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649046 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.648985    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-envoy\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649403 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.649248    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-socket\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649403 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.649289    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-certs\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649403 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.649369    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-credential-socket\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649619 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.649597    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-data\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.649662 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.649637    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.651577 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.651555    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-envoy\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.651657 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.651641    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-podinfo\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.673690 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.673655    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjs5\" (UniqueName: \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-kube-api-access-lkjs5\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.674756 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.674736    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-token\") pod \"router-gateway-1-openshift-default-65d54ccdd5-knxbd\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:16.782791 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:16.782755    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:17.111890 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:17.111846    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"]
Oct 30 03:25:17.112056 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:25:17.111964    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bd5c5a_07ab_4ebe_a29a_a54927b2aabb.slice/crio-c4c27b1bbd1020181a9c5518bfaa94e8f7bc6625c629081f14447905411768d8 WatchSource:0}: Error finding container c4c27b1bbd1020181a9c5518bfaa94e8f7bc6625c629081f14447905411768d8: Status 404 returned error can't find the container with id c4c27b1bbd1020181a9c5518bfaa94e8f7bc6625c629081f14447905411768d8
Oct 30 03:25:17.114147 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:17.114116    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:25:17.114238 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:17.114183    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:25:17.114238 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:17.114214    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:25:17.435223 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:17.435130    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd" event={"ID":"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb","Type":"ContainerStarted","Data":"60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51"}
Oct 30 03:25:17.435223 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:17.435192    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd" event={"ID":"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb","Type":"ContainerStarted","Data":"c4c27b1bbd1020181a9c5518bfaa94e8f7bc6625c629081f14447905411768d8"}
Oct 30 03:25:17.480918 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:17.480852    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd" podStartSLOduration=1.480840113 podStartE2EDuration="1.480840113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:17.479653556 +0000 UTC m=+1781.685833775" watchObservedRunningTime="2026-10-30 03:25:17.480840113 +0000 UTC m=+1781.687020331"
Oct 30 03:25:17.783266 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:17.783235    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:18.788801 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:18.788763    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:19.445707 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:19.445671    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:19.446732 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:19.446711    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:20.717284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.717249    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"]
Oct 30 03:25:20.720831 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.720808    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.750355 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.750315    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Oct 30 03:25:20.750724 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.750707    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-qvsjf\""
Oct 30 03:25:20.751759 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.751739    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"]
Oct 30 03:25:20.784990 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.784950    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4j6\" (UniqueName: \"kubernetes.io/projected/06eea531-8967-49ad-b984-4de9d7f2feba-kube-api-access-2n4j6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.785167 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.785078    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.785216 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.785168    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.785255 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.785218    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.785300 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.785259    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.785345 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.785328    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.886214 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.886116    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.886214 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.886158    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.886214 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.886181    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.886214 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.886205    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.886526 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.886224    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.886526 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.886287    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4j6\" (UniqueName: \"kubernetes.io/projected/06eea531-8967-49ad-b984-4de9d7f2feba-kube-api-access-2n4j6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.886679 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.886653    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.886748 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.886682    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.886748 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.886737    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.889220 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.889197    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.889444 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.889423    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:20.910349 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:20.910316    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4j6\" (UniqueName: \"kubernetes.io/projected/06eea531-8967-49ad-b984-4de9d7f2feba-kube-api-access-2n4j6\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:21.035975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:21.035942    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:21.335004 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:25:21.334962    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06eea531_8967_49ad_b984_4de9d7f2feba.slice/crio-77651c06f3a17da794b07244903ae1a6394613c05096974a78886ef248958f97 WatchSource:0}: Error finding container 77651c06f3a17da794b07244903ae1a6394613c05096974a78886ef248958f97: Status 404 returned error can't find the container with id 77651c06f3a17da794b07244903ae1a6394613c05096974a78886ef248958f97
Oct 30 03:25:21.335004 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:21.334969    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"]
Oct 30 03:25:21.457542 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:21.457503    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" event={"ID":"06eea531-8967-49ad-b984-4de9d7f2feba","Type":"ContainerStarted","Data":"d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31"}
Oct 30 03:25:21.457714 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:21.457568    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" event={"ID":"06eea531-8967-49ad-b984-4de9d7f2feba","Type":"ContainerStarted","Data":"77651c06f3a17da794b07244903ae1a6394613c05096974a78886ef248958f97"}
Oct 30 03:25:21.479735 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:21.479705    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"]
Oct 30 03:25:21.513479 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:21.513421    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" podStartSLOduration=1.513406971 podStartE2EDuration="1.513406971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:21.51233218 +0000 UTC m=+1785.718512395" watchObservedRunningTime="2026-10-30 03:25:21.513406971 +0000 UTC m=+1785.719587189"
Oct 30 03:25:21.593164 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:21.593040    4494 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Oct 30 03:25:21.593164 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:21.593126    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs podName:06eea531-8967-49ad-b984-4de9d7f2feba nodeName:}" failed. No retries permitted until 2026-10-30 03:25:22.093102976 +0000 UTC m=+1786.299283192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" (UID: "06eea531-8967-49ad-b984-4de9d7f2feba") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Oct 30 03:25:22.015131 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.015037    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"]
Oct 30 03:25:22.019157 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.019128    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.035637 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.035613    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Oct 30 03:25:22.035769 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.035749    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Oct 30 03:25:22.035824 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.035808    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Oct 30 03:25:22.035924 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.035875    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-4sgjv\""
Oct 30 03:25:22.036873 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.036860    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:22.049711 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.049687    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"]
Oct 30 03:25:22.098186 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.098148    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-socket\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.098362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.098286    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.098362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.098346    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-token\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.098463 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.098384    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-certs\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.098463 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:22.098386    4494 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Oct 30 03:25:22.098545 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:22.098465    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs podName:06eea531-8967-49ad-b984-4de9d7f2feba nodeName:}" failed. No retries permitted until 2026-10-30 03:25:23.098451976 +0000 UTC m=+1787.304632171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" (UID: "06eea531-8967-49ad-b984-4de9d7f2feba") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Oct 30 03:25:22.098545 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.098504    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-data\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.098545 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.098520    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6pwn\" (UniqueName: \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-kube-api-access-d6pwn\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.098545 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.098539    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-credential-socket\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.098705 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.098564    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-podinfo\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.098705 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.098674    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-envoy\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.199866 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.199826    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-certs\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.199883    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-data\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.199932    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6pwn\" (UniqueName: \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-kube-api-access-d6pwn\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.199949    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-credential-socket\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.199976    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-podinfo\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.200000    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-envoy\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200372 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.200126    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-socket\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200372 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.200261    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200372 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.200321    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-token\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200372 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.200323    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-certs\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200556 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.200392    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-credential-socket\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200556 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.200521    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-socket\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.200652 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.200569    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-data\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.201259 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.201236    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.202484 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.202453    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-envoy\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.202728 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.202708    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-podinfo\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.224199 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.224173    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-token\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.224460 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.224438    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6pwn\" (UniqueName: \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-kube-api-access-d6pwn\") pod \"router-gateway-1-openshift-default-777b5964f-xhlv9\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.334412 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.334376    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:22.676829 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:25:22.676801    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295bc224_25b3_45e7_9ef1_6c97a2ce5eaf.slice/crio-0efd2296ce91ffa81d91ef031bd8e9f00ef18c428004190f3a708a171236aa33 WatchSource:0}: Error finding container 0efd2296ce91ffa81d91ef031bd8e9f00ef18c428004190f3a708a171236aa33: Status 404 returned error can't find the container with id 0efd2296ce91ffa81d91ef031bd8e9f00ef18c428004190f3a708a171236aa33
Oct 30 03:25:22.677084 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.677061    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"]
Oct 30 03:25:22.678998 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.678958    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:25:22.679106 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.679036    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:25:22.679106 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:22.679070    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:25:23.109327 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:23.109276    4494 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Oct 30 03:25:23.109771 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:23.109367    4494 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs podName:06eea531-8967-49ad-b984-4de9d7f2feba nodeName:}" failed. No retries permitted until 2026-10-30 03:25:25.109354026 +0000 UTC m=+1789.315534222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" (UID: "06eea531-8967-49ad-b984-4de9d7f2feba") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Oct 30 03:25:23.469364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.469262    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9" event={"ID":"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf","Type":"ContainerStarted","Data":"fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba"}
Oct 30 03:25:23.469364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.469325    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9" event={"ID":"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf","Type":"ContainerStarted","Data":"0efd2296ce91ffa81d91ef031bd8e9f00ef18c428004190f3a708a171236aa33"}
Oct 30 03:25:23.479054 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:23.479013    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31" cmd=["/bin/sleep","15"]
Oct 30 03:25:23.479223 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.479074    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:25:23.479223 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:23.479090    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" podUID="06eea531-8967-49ad-b984-4de9d7f2feba" containerName="main" containerID="cri-o://d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31"
Oct 30 03:25:23.479223 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.479136    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" podUID="06eea531-8967-49ad-b984-4de9d7f2feba" containerName="main" containerID="cri-o://d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31" gracePeriod=60
Oct 30 03:25:23.556641 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.556592    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9" podStartSLOduration=2.556577812 podStartE2EDuration="2.556577812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:23.555831684 +0000 UTC m=+1787.762011904" watchObservedRunningTime="2026-10-30 03:25:23.556577812 +0000 UTC m=+1787.762758090"
Oct 30 03:25:23.728539 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.728477    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:23.815922 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.815870    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-dshm\") pod \"06eea531-8967-49ad-b984-4de9d7f2feba\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") "
Oct 30 03:25:23.816093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.815940    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-home\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-home\") pod \"06eea531-8967-49ad-b984-4de9d7f2feba\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") "
Oct 30 03:25:23.816093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.815986    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-tmp-dir\") pod \"06eea531-8967-49ad-b984-4de9d7f2feba\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") "
Oct 30 03:25:23.816093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.816032    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-model-cache\") pod \"06eea531-8967-49ad-b984-4de9d7f2feba\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") "
Oct 30 03:25:23.816093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.816088    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs\" (UniqueName: \"kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs\") pod \"06eea531-8967-49ad-b984-4de9d7f2feba\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") "
Oct 30 03:25:23.816284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.816168    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06eea531-8967-49ad-b984-4de9d7f2feba-kube-api-access-2n4j6\" (UniqueName: \"kubernetes.io/projected/06eea531-8967-49ad-b984-4de9d7f2feba-kube-api-access-2n4j6\") pod \"06eea531-8967-49ad-b984-4de9d7f2feba\" (UID: \"06eea531-8967-49ad-b984-4de9d7f2feba\") "
Oct 30 03:25:23.816284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.816216    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-home" pod "06eea531-8967-49ad-b984-4de9d7f2feba" (UID: "06eea531-8967-49ad-b984-4de9d7f2feba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:23.816396 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.816317    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-model-cache" pod "06eea531-8967-49ad-b984-4de9d7f2feba" (UID: "06eea531-8967-49ad-b984-4de9d7f2feba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:23.816396 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.816324    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-tmp-dir" pod "06eea531-8967-49ad-b984-4de9d7f2feba" (UID: "06eea531-8967-49ad-b984-4de9d7f2feba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:23.816501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.816474    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:23.816501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.816488    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:23.816501 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.816498    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:23.818314 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.818289    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-dshm" pod "06eea531-8967-49ad-b984-4de9d7f2feba" (UID: "06eea531-8967-49ad-b984-4de9d7f2feba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:23.818444 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.818351    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs" pod "06eea531-8967-49ad-b984-4de9d7f2feba" (UID: "06eea531-8967-49ad-b984-4de9d7f2feba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:23.818491 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.818449    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06eea531-8967-49ad-b984-4de9d7f2feba-kube-api-access-2n4j6" pod "06eea531-8967-49ad-b984-4de9d7f2feba" (UID: "06eea531-8967-49ad-b984-4de9d7f2feba"). InnerVolumeSpecName "kube-api-access-2n4j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:23.917669 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.917628    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06eea531-8967-49ad-b984-4de9d7f2feba-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:23.917669 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.917662    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4j6\" (UniqueName: \"kubernetes.io/projected/06eea531-8967-49ad-b984-4de9d7f2feba-kube-api-access-2n4j6\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:23.917888 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:23.917673    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06eea531-8967-49ad-b984-4de9d7f2feba-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:24.083211 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:24.083163    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:25:24.334655 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.334555    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:24.339665 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.339641    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:24.475843 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.475810    4494 generic.go:356] "Generic (PLEG): container finished" podID="06eea531-8967-49ad-b984-4de9d7f2feba" containerID="d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31" exitCode=0
Oct 30 03:25:24.476048 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.475861    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"
Oct 30 03:25:24.476048 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.475888    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" event={"ID":"06eea531-8967-49ad-b984-4de9d7f2feba","Type":"ContainerDied","Data":"d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31"}
Oct 30 03:25:24.476048 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.475975    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl" event={"ID":"06eea531-8967-49ad-b984-4de9d7f2feba","Type":"ContainerDied","Data":"77651c06f3a17da794b07244903ae1a6394613c05096974a78886ef248958f97"}
Oct 30 03:25:24.476048 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.476000    4494 scope.go:122] "RemoveContainer" containerID="d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31"
Oct 30 03:25:24.476370 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.476350    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:24.477302 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.477283    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:24.485340 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.485153    4494 scope.go:122] "RemoveContainer" containerID="d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31"
Oct 30 03:25:24.485486 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:24.485465    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31\": container with ID starting with d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31 not found: ID does not exist" containerID="d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31"
Oct 30 03:25:24.485551 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.485498    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31"} err="failed to get container status \"d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31\": rpc error: code = NotFound desc = could not find container \"d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31\": container with ID starting with d4cf8a83684b26641173b168aa910ade29a6569b0901b0b2894b603fb777ba31 not found: ID does not exist"
Oct 30 03:25:24.574602 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.574561    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"]
Oct 30 03:25:24.578085 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:24.578055    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-mxlcl"]
Oct 30 03:25:25.927103 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:25.927071    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06eea531-8967-49ad-b984-4de9d7f2feba" path="/var/lib/kubelet/pods/06eea531-8967-49ad-b984-4de9d7f2feba/volumes"
Oct 30 03:25:26.423339 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:26.423307    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"]
Oct 30 03:25:26.423540 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:26.423514    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd" podUID="a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" containerName="istio-proxy" containerID="cri-o://60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51" gracePeriod=30
Oct 30 03:25:27.727184 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.727148    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"]
Oct 30 03:25:27.727718 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.727702    4494 cpu_manager.go:414] "containerMap: removing container" podUID="06eea531-8967-49ad-b984-4de9d7f2feba" containerName="main"
Oct 30 03:25:27.727762 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.727722    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06eea531-8967-49ad-b984-4de9d7f2feba" containerName="main"
Oct 30 03:25:27.727872 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.727861    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="06eea531-8967-49ad-b984-4de9d7f2feba" containerName="main"
Oct 30 03:25:27.731172 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.731146    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.761197 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.761164    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Oct 30 03:25:27.761400 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.761260    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-t9vdl\""
Oct 30 03:25:27.762856 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.762829    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"]
Oct 30 03:25:27.853781 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.853742    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvdq5\" (UniqueName: \"kubernetes.io/projected/20f94527-6892-4ca7-9dc2-fecbb4172f97-kube-api-access-cvdq5\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.854006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.853786    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.854006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.853884    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.854006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.853953    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f94527-6892-4ca7-9dc2-fecbb4172f97-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.854159 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.854045    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.854159 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.854075    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.955155 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.955114    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvdq5\" (UniqueName: \"kubernetes.io/projected/20f94527-6892-4ca7-9dc2-fecbb4172f97-kube-api-access-cvdq5\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.955155 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.955156    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.955401 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.955199    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.955401 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.955230    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f94527-6892-4ca7-9dc2-fecbb4172f97-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.955401 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.955261    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.955401 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.955290    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.955632 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.955605    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.955683 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.955666    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.955742 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.955725    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.957767 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.957752    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.957876 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.957860    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f94527-6892-4ca7-9dc2-fecbb4172f97-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:27.978658 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:27.978586    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvdq5\" (UniqueName: \"kubernetes.io/projected/20f94527-6892-4ca7-9dc2-fecbb4172f97-kube-api-access-cvdq5\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:28.045922 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:28.045869    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:28.344429 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:28.344399    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"]
Oct 30 03:25:28.344545 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:25:28.344510    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f94527_6892_4ca7_9dc2_fecbb4172f97.slice/crio-d4bff679915386f51c1cd239fe3b8e82292f984a36076ca3ff90bd000bb236fd WatchSource:0}: Error finding container d4bff679915386f51c1cd239fe3b8e82292f984a36076ca3ff90bd000bb236fd: Status 404 returned error can't find the container with id d4bff679915386f51c1cd239fe3b8e82292f984a36076ca3ff90bd000bb236fd
Oct 30 03:25:28.497544 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:28.497504    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg" event={"ID":"20f94527-6892-4ca7-9dc2-fecbb4172f97","Type":"ContainerStarted","Data":"1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42"}
Oct 30 03:25:28.497685 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:28.497570    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg" event={"ID":"20f94527-6892-4ca7-9dc2-fecbb4172f97","Type":"ContainerStarted","Data":"d4bff679915386f51c1cd239fe3b8e82292f984a36076ca3ff90bd000bb236fd"}
Oct 30 03:25:28.539597 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:28.539542    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg" podStartSLOduration=1.5395270220000001 podStartE2EDuration="1.539527022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:28.538803468 +0000 UTC m=+1792.744983711" watchObservedRunningTime="2026-10-30 03:25:28.539527022 +0000 UTC m=+1792.745707241"
Oct 30 03:25:29.046479 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:29.046430    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:31.714509 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.714442    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:31.810518 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.810458    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-token\" (UniqueName: \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-token\") pod \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") "
Oct 30 03:25:31.810712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.810537    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-kube-api-access-lkjs5\" (UniqueName: \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-kube-api-access-lkjs5\") pod \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") "
Oct 30 03:25:31.810712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.810605    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-socket\") pod \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") "
Oct 30 03:25:31.810712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.810636    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-envoy\") pod \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") "
Oct 30 03:25:31.810937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.810718    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-data\") pod \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") "
Oct 30 03:25:31.810937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.810764    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-podinfo\") pod \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") "
Oct 30 03:25:31.810937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.810800    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-credential-socket\") pod \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") "
Oct 30 03:25:31.810937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.810869    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istiod-ca-cert\") pod \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") "
Oct 30 03:25:31.811149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.811021    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-certs\") pod \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\" (UID: \"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb\") "
Oct 30 03:25:31.811531 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.811503    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-certs" pod "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" (UID: "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:31.812014 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.811982    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-credential-socket" pod "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" (UID: "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:31.812116 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.812056    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-socket" pod "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" (UID: "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:31.812116 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.812061    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istiod-ca-cert" pod "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" (UID: "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:31.812116 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.812108    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-data" pod "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" (UID: "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:31.813754 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.813723    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-token" pod "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" (UID: "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:31.813927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.813874    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-envoy" pod "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" (UID: "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:31.814044 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.813983    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-kube-api-access-lkjs5" pod "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" (UID: "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb"). InnerVolumeSpecName "kube-api-access-lkjs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:31.814736 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.814712    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-podinfo" pod "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" (UID: "a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:25:31.912458 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.912416    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:31.912458 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.912450    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkjs5\" (UniqueName: \"kubernetes.io/projected/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-kube-api-access-lkjs5\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:31.912458 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.912461    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:31.912458 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.912470    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:31.912739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.912478    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:31.912739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.912486    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:31.912739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.912493    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:31.912739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.912501    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:31.912739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:31.912509    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:32.522480 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.522448    4494 generic.go:356] "Generic (PLEG): container finished" podID="a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" containerID="60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51" exitCode=0
Oct 30 03:25:32.522651 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.522498    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"
Oct 30 03:25:32.522651 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.522515    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd" event={"ID":"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb","Type":"ContainerDied","Data":"60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51"}
Oct 30 03:25:32.522651 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.522560    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd" event={"ID":"a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb","Type":"ContainerDied","Data":"c4c27b1bbd1020181a9c5518bfaa94e8f7bc6625c629081f14447905411768d8"}
Oct 30 03:25:32.522651 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.522583    4494 scope.go:122] "RemoveContainer" containerID="60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51"
Oct 30 03:25:32.531334 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.531311    4494 scope.go:122] "RemoveContainer" containerID="60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51"
Oct 30 03:25:32.531605 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:32.531584    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51\": container with ID starting with 60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51 not found: ID does not exist" containerID="60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51"
Oct 30 03:25:32.531664 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.531614    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51"} err="failed to get container status \"60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51\": rpc error: code = NotFound desc = could not find container \"60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51\": container with ID starting with 60eed71cec81fc684b19c4ccdc1c98c70333b45188d2ffc514b3989f304f2a51 not found: ID does not exist"
Oct 30 03:25:32.573919 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.573872    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"]
Oct 30 03:25:32.602925 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.601887    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-65d54ccdd5-knxbd"]
Oct 30 03:25:32.967464 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.967433    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"]
Oct 30 03:25:32.976663 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:32.976630    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42" cmd=["/bin/sleep","15"]
Oct 30 03:25:32.976788 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.976675    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:25:32.976788 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:32.976689    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg" podUID="20f94527-6892-4ca7-9dc2-fecbb4172f97" containerName="main" containerID="cri-o://1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42"
Oct 30 03:25:32.976788 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:32.976728    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg" podUID="20f94527-6892-4ca7-9dc2-fecbb4172f97" containerName="main" containerID="cri-o://1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42" gracePeriod=60
Oct 30 03:25:33.276918 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.276877    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:33.427674 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.427641    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-model-cache\") pod \"20f94527-6892-4ca7-9dc2-fecbb4172f97\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") "
Oct 30 03:25:33.427674 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.427689    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f94527-6892-4ca7-9dc2-fecbb4172f97-tls-certs\" (UniqueName: \"kubernetes.io/secret/20f94527-6892-4ca7-9dc2-fecbb4172f97-tls-certs\") pod \"20f94527-6892-4ca7-9dc2-fecbb4172f97\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") "
Oct 30 03:25:33.428029 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.427735    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f94527-6892-4ca7-9dc2-fecbb4172f97-kube-api-access-cvdq5\" (UniqueName: \"kubernetes.io/projected/20f94527-6892-4ca7-9dc2-fecbb4172f97-kube-api-access-cvdq5\") pod \"20f94527-6892-4ca7-9dc2-fecbb4172f97\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") "
Oct 30 03:25:33.428029 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.427810    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-dshm\") pod \"20f94527-6892-4ca7-9dc2-fecbb4172f97\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") "
Oct 30 03:25:33.428029 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.427841    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-home\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-home\") pod \"20f94527-6892-4ca7-9dc2-fecbb4172f97\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") "
Oct 30 03:25:33.428029 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.427881    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-tmp-dir\") pod \"20f94527-6892-4ca7-9dc2-fecbb4172f97\" (UID: \"20f94527-6892-4ca7-9dc2-fecbb4172f97\") "
Oct 30 03:25:33.428029 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.427970    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-model-cache" pod "20f94527-6892-4ca7-9dc2-fecbb4172f97" (UID: "20f94527-6892-4ca7-9dc2-fecbb4172f97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:33.428313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.428111    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-home" pod "20f94527-6892-4ca7-9dc2-fecbb4172f97" (UID: "20f94527-6892-4ca7-9dc2-fecbb4172f97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:33.428313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.428223    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:33.428313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.428240    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:33.428313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.428274    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-tmp-dir" pod "20f94527-6892-4ca7-9dc2-fecbb4172f97" (UID: "20f94527-6892-4ca7-9dc2-fecbb4172f97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:33.430188 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.430166    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-dshm" pod "20f94527-6892-4ca7-9dc2-fecbb4172f97" (UID: "20f94527-6892-4ca7-9dc2-fecbb4172f97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:33.430638 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.430615    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f94527-6892-4ca7-9dc2-fecbb4172f97-kube-api-access-cvdq5" pod "20f94527-6892-4ca7-9dc2-fecbb4172f97" (UID: "20f94527-6892-4ca7-9dc2-fecbb4172f97"). InnerVolumeSpecName "kube-api-access-cvdq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:33.430715 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.430616    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f94527-6892-4ca7-9dc2-fecbb4172f97-tls-certs" pod "20f94527-6892-4ca7-9dc2-fecbb4172f97" (UID: "20f94527-6892-4ca7-9dc2-fecbb4172f97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:25:33.528740 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.528709    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f94527-6892-4ca7-9dc2-fecbb4172f97-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:33.528740 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.528740    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvdq5\" (UniqueName: \"kubernetes.io/projected/20f94527-6892-4ca7-9dc2-fecbb4172f97-kube-api-access-cvdq5\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:33.529015 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.528753    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:33.529015 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.528766    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f94527-6892-4ca7-9dc2-fecbb4172f97-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:33.529278 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.529254    4494 generic.go:356] "Generic (PLEG): container finished" podID="20f94527-6892-4ca7-9dc2-fecbb4172f97" containerID="1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42" exitCode=0
Oct 30 03:25:33.529354 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.529318    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"
Oct 30 03:25:33.529354 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.529337    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg" event={"ID":"20f94527-6892-4ca7-9dc2-fecbb4172f97","Type":"ContainerDied","Data":"1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42"}
Oct 30 03:25:33.529463 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.529388    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg" event={"ID":"20f94527-6892-4ca7-9dc2-fecbb4172f97","Type":"ContainerDied","Data":"d4bff679915386f51c1cd239fe3b8e82292f984a36076ca3ff90bd000bb236fd"}
Oct 30 03:25:33.529463 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.529436    4494 scope.go:122] "RemoveContainer" containerID="1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42"
Oct 30 03:25:33.541175 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.541151    4494 scope.go:122] "RemoveContainer" containerID="1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42"
Oct 30 03:25:33.541558 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:33.541531    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42\": container with ID starting with 1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42 not found: ID does not exist" containerID="1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42"
Oct 30 03:25:33.541665 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.541568    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42"} err="failed to get container status \"1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42\": rpc error: code = NotFound desc = could not find container \"1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42\": container with ID starting with 1a7b8d70c0c0318dd17d2a122e4569ddb94db9803affd59a9ac2615e90927a42 not found: ID does not exist"
Oct 30 03:25:33.603927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.580603    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"]
Oct 30 03:25:33.605986 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.605953    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-8g5vg"]
Oct 30 03:25:33.927505 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.927415    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f94527-6892-4ca7-9dc2-fecbb4172f97" path="/var/lib/kubelet/pods/20f94527-6892-4ca7-9dc2-fecbb4172f97/volumes"
Oct 30 03:25:33.928000 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:33.927969    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" path="/var/lib/kubelet/pods/a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb/volumes"
Oct 30 03:25:34.296366 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:34.295397    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"]
Oct 30 03:25:34.296366 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:34.295728    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9" podUID="295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" containerName="istio-proxy" containerID="cri-o://fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba" gracePeriod=30
Oct 30 03:25:37.005330 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.005268    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"]
Oct 30 03:25:37.005892 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.005832    4494 cpu_manager.go:414] "containerMap: removing container" podUID="a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" containerName="istio-proxy"
Oct 30 03:25:37.005892 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.005848    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" containerName="istio-proxy"
Oct 30 03:25:37.005892 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.005873    4494 cpu_manager.go:414] "containerMap: removing container" podUID="20f94527-6892-4ca7-9dc2-fecbb4172f97" containerName="main"
Oct 30 03:25:37.005892 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.005883    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f94527-6892-4ca7-9dc2-fecbb4172f97" containerName="main"
Oct 30 03:25:37.006148 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.006018    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f94527-6892-4ca7-9dc2-fecbb4172f97" containerName="main"
Oct 30 03:25:37.006148 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.006040    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6bd5c5a-07ab-4ebe-a29a-a54927b2aabb" containerName="istio-proxy"
Oct 30 03:25:37.011397 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.011370    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.025858 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.025826    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"]
Oct 30 03:25:37.036582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.036541    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Oct 30 03:25:37.036582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.036561    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Oct 30 03:25:37.036582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.036565    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Oct 30 03:25:37.036887 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.036616    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:25:37.036887 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.036650    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-nf54q\""
Oct 30 03:25:37.036887 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.036723    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Oct 30 03:25:37.164388 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.164349    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-kserve-provision-location\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.164592 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.164399    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa138a32-4b5a-45a6-96c3-051dea2bd937-cabundle-cert\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.164592 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.164472    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa138a32-4b5a-45a6-96c3-051dea2bd937-tls-certs\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.164592 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.164508    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-model-cache\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.164592 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.164536    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-tmp-dir\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.164841 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.164585    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-home\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.164841 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.164652    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-dshm\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.164841 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.164682    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxw2\" (UniqueName: \"kubernetes.io/projected/aa138a32-4b5a-45a6-96c3-051dea2bd937-kube-api-access-xhxw2\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.266251 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.266167    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-kserve-provision-location\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.266251 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.266237    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa138a32-4b5a-45a6-96c3-051dea2bd937-cabundle-cert\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.266478 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.266270    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa138a32-4b5a-45a6-96c3-051dea2bd937-tls-certs\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.266478 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.266297    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-model-cache\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.266478 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.266328    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-tmp-dir\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.266478 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.266353    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-home\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.266478 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.266394    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-dshm\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.266478 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.266433    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxw2\" (UniqueName: \"kubernetes.io/projected/aa138a32-4b5a-45a6-96c3-051dea2bd937-kube-api-access-xhxw2\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.266746 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.266695    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-model-cache\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.267036 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.267006    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-kserve-provision-location\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.267036 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.267031    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-tmp-dir\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.267240 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.267159    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa138a32-4b5a-45a6-96c3-051dea2bd937-cabundle-cert\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.267240 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.267173    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-home\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.269073 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.269046    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-dshm\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.269215 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.269196    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa138a32-4b5a-45a6-96c3-051dea2bd937-tls-certs\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.289989 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.289961    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxw2\" (UniqueName: \"kubernetes.io/projected/aa138a32-4b5a-45a6-96c3-051dea2bd937-kube-api-access-xhxw2\") pod \"auth-enabled-test-kserve-65f985fb54-spkrb\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.329765 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.329732    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:25:37.652527 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.652489    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"]
Oct 30 03:25:37.752776 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:37.752735    4494 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295bc224_25b3_45e7_9ef1_6c97a2ce5eaf.slice/crio-fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295bc224_25b3_45e7_9ef1_6c97a2ce5eaf.slice/crio-conmon-fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@1-18643-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 03:25:37.846177 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.846150    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:37.973433 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.973393    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istiod-ca-cert\") pod \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") "
Oct 30 03:25:37.973630 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.973446    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-socket\") pod \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") "
Oct 30 03:25:37.973630 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.973494    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-podinfo\") pod \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") "
Oct 30 03:25:37.973630 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.973565    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-envoy\") pod \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") "
Oct 30 03:25:37.973630 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.973623    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-credential-socket\") pod \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") "
Oct 30 03:25:37.973891 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.973657    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-certs\") pod \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") "
Oct 30 03:25:37.973891 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.973768    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-data\") pod \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") "
Oct 30 03:25:37.973891 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.973807    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istiod-ca-cert" pod "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" (UID: "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:25:37.973891 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.973827    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-token\" (UniqueName: \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-token\") pod \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") "
Oct 30 03:25:37.973891 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.973866    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-kube-api-access-d6pwn\" (UniqueName: \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-kube-api-access-d6pwn\") pod \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\" (UID: \"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf\") "
Oct 30 03:25:37.974211 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.974183    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:37.974211 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.974195    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-credential-socket" pod "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" (UID: "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:37.974380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.974345    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-data" pod "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" (UID: "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:37.974380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.974374    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-certs" pod "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" (UID: "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:37.974717 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.974446    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-socket" pod "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" (UID: "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:37.976241 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.976211    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-kube-api-access-d6pwn" pod "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" (UID: "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf"). InnerVolumeSpecName "kube-api-access-d6pwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:37.976373 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.976239    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-envoy" pod "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" (UID: "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:25:37.976601 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.976583    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-podinfo" pod "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" (UID: "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:25:37.976703 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:37.976601    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-token" pod "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" (UID: "295bc224-25b3-45e7-9ef1-6c97a2ce5eaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:25:38.075354 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.075307    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:38.075354 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.075342    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:38.075354 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.075353    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6pwn\" (UniqueName: \"kubernetes.io/projected/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-kube-api-access-d6pwn\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:38.075354 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.075362    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:38.076098 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.075372    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:38.076098 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.075380    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:38.076098 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.075388    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:38.076098 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.075397    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:25:38.556575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.556464    4494 generic.go:356] "Generic (PLEG): container finished" podID="295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" containerID="fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba" exitCode=0
Oct 30 03:25:38.556575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.556519    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"
Oct 30 03:25:38.556575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.556561    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9" event={"ID":"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf","Type":"ContainerDied","Data":"fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba"}
Oct 30 03:25:38.556867 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.556633    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9" event={"ID":"295bc224-25b3-45e7-9ef1-6c97a2ce5eaf","Type":"ContainerDied","Data":"0efd2296ce91ffa81d91ef031bd8e9f00ef18c428004190f3a708a171236aa33"}
Oct 30 03:25:38.556867 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.556657    4494 scope.go:122] "RemoveContainer" containerID="fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba"
Oct 30 03:25:38.560937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.559208    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" event={"ID":"aa138a32-4b5a-45a6-96c3-051dea2bd937","Type":"ContainerStarted","Data":"0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7"}
Oct 30 03:25:38.560937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.559261    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" event={"ID":"aa138a32-4b5a-45a6-96c3-051dea2bd937","Type":"ContainerStarted","Data":"cf1747296a5544561bdb106802920afd5c8ce32e218a3febb820305b6f5b5dd9"}
Oct 30 03:25:38.612927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.608163    4494 scope.go:122] "RemoveContainer" containerID="fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba"
Oct 30 03:25:38.612927 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:38.608669    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba\": container with ID starting with fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba not found: ID does not exist" containerID="fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba"
Oct 30 03:25:38.612927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.608708    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba"} err="failed to get container status \"fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba\": rpc error: code = NotFound desc = could not find container \"fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba\": container with ID starting with fb96fe117561c467854bbd6e18632734bc5c7bdf64b0ab90601bec3a0831c7ba not found: ID does not exist"
Oct 30 03:25:38.677306 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.677265    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"]
Oct 30 03:25:38.694369 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:38.681126    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-777b5964f-xhlv9"]
Oct 30 03:25:39.939721 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:39.939686    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" path="/var/lib/kubelet/pods/295bc224-25b3-45e7-9ef1-6c97a2ce5eaf/volumes"
Oct 30 03:25:41.574000 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:41.573969    4494 generic.go:356] "Generic (PLEG): container finished" podID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerID="0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7" exitCode=0
Oct 30 03:25:41.574495 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:41.574041    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" event={"ID":"aa138a32-4b5a-45a6-96c3-051dea2bd937","Type":"ContainerDied","Data":"0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7"}
Oct 30 03:25:41.575544 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:41.575528    4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:25:54.086224 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:25:54.086183    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:25:59.844114 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:59.844069    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"]
Oct 30 03:25:59.844674 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:59.844645    4494 cpu_manager.go:414] "containerMap: removing container" podUID="295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" containerName="istio-proxy"
Oct 30 03:25:59.844674 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:59.844673    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" containerName="istio-proxy"
Oct 30 03:25:59.844924 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:59.844779    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="295bc224-25b3-45e7-9ef1-6c97a2ce5eaf" containerName="istio-proxy"
Oct 30 03:25:59.941349 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:59.941315    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"]
Oct 30 03:25:59.941534 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:25:59.941455    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.042012 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.041971    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.042198 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.042035    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.042198 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.042078    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.042290 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.042210    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.042290 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.042272    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.042375 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.042304    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.042375 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.042340    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.042456 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.042397    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89kn\" (UniqueName: \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-kube-api-access-w89kn\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.042496 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.042464    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.143645 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.143550    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.143645 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.143587    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.143645 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.143623    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.143948 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.143654    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89kn\" (UniqueName: \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-kube-api-access-w89kn\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.143995 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.143967    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.144059 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.144039    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.144123 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.144082    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.144177 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.144129    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.144933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.144242    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.144933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.144283    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.144933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.144572    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.144933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.144710    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.144933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.144800    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.145257 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.144940    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.146856 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.146834    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.147056 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.147040    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.168342 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.168314    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89kn\" (UniqueName: \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-kube-api-access-w89kn\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.168883 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.168854    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.302705 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.302669    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:00.647671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.647636    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"]
Oct 30 03:26:00.648135 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:26:00.648104    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad55ffb_b3f3_4d4c_946c_d13300b8dab4.slice/crio-3b81502dfffe2bac39dc1e21ee573b2bab14d0d987424a0b63c113fea336620c WatchSource:0}: Error finding container 3b81502dfffe2bac39dc1e21ee573b2bab14d0d987424a0b63c113fea336620c: Status 404 returned error can't find the container with id 3b81502dfffe2bac39dc1e21ee573b2bab14d0d987424a0b63c113fea336620c
Oct 30 03:26:00.650346 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.650311    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:26:00.650460 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.650399    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:26:00.650460 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.650433    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:26:00.727819 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:00.727780    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" event={"ID":"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4","Type":"ContainerStarted","Data":"3b81502dfffe2bac39dc1e21ee573b2bab14d0d987424a0b63c113fea336620c"}
Oct 30 03:26:01.734775 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:01.734731    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" event={"ID":"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4","Type":"ContainerStarted","Data":"7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3"}
Oct 30 03:26:01.809043 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:01.808981    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" podStartSLOduration=2.808965872 podStartE2EDuration="2.808965872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:01.795398254 +0000 UTC m=+1826.001578473" watchObservedRunningTime="2026-10-30 03:26:01.808965872 +0000 UTC m=+1826.015146089"
Oct 30 03:26:02.304194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:02.304151    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:02.305744 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:02.305709    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 03:26:02.305928 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:02.305769    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" podUID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 03:26:03.304349 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:03.304308    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 03:26:03.304861 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:03.304371    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" podUID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 03:26:04.304694 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:04.304287    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 03:26:04.304694 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:04.304357    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" podUID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 03:26:05.303836 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:05.303376    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 03:26:05.303836 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:05.303449    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" podUID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 03:26:06.304801 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:06.304375    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 30 03:26:06.304801 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:06.304449    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" podUID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 30 03:26:07.308317 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:07.308278    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:07.308828 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:07.308794    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:07.309598 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:07.309576    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:26:12.791442 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:12.791403    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" event={"ID":"aa138a32-4b5a-45a6-96c3-051dea2bd937","Type":"ContainerStarted","Data":"4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da"}
Oct 30 03:26:12.863767 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:12.863691    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podStartSLOduration=3.8124580569999997 podStartE2EDuration="36.863673296s" totalImagesPullingTime="30.279957673s" totalInitContainerRuntime="2.771257566s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:12.850742555 +0000 UTC m=+1837.056922777" watchObservedRunningTime="2026-10-30 03:26:12.863673296 +0000 UTC m=+1837.069853516"
Oct 30 03:26:13.330261 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:13.330221    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:26:17.330134 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:17.330036    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:26:17.331605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:17.331574    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:26:24.084522 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:26:24.084485    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:26:27.330264 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:27.330222    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:26:37.330447 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:37.330400    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:26:47.330702 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:47.330660    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:26:54.083739 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:26:54.083699    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:26:57.330174 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:26:57.330126    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:27:07.330724 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:07.330668    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:27:17.331275 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:17.331227    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:27:24.081961 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:27:24.081880    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:27:27.330556 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:27.330508    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:27:37.331120 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:37.331072    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:27:45.727645 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.727609    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"]
Oct 30 03:27:45.731187 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.731165    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.747601 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.747551    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Oct 30 03:27:45.747741 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.747633    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-gsg7j\""
Oct 30 03:27:45.763707 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.763674    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"]
Oct 30 03:27:45.804368 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.804328    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.804368 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.804368    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.804597 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.804405    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.804597 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.804542    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.804597 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.804573    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.804720 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.804603    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fln\" (UniqueName: \"kubernetes.io/projected/3dfbdd36-8e2c-41d9-927b-510e1d9747af-kube-api-access-j2fln\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.836692 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.836656    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Oct 30 03:27:45.836862 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.836728    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Oct 30 03:27:45.905770 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.905737    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.905770 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.905775    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.906037 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.905797    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fln\" (UniqueName: \"kubernetes.io/projected/3dfbdd36-8e2c-41d9-927b-510e1d9747af-kube-api-access-j2fln\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.906037 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.905943    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.906037 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.905987    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.906192 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.906048    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.906261 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.906244    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.906349 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.906328    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.906411 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.906379    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.908533 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.908505    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.908925 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.908878    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:45.933950 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:45.930004    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fln\" (UniqueName: \"kubernetes.io/projected/3dfbdd36-8e2c-41d9-927b-510e1d9747af-kube-api-access-j2fln\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:46.045790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:46.045755    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:46.354012 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:46.353974    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"]
Oct 30 03:27:46.354175 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:27:46.354149    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfbdd36_8e2c_41d9_927b_510e1d9747af.slice/crio-050910dae45ef4e0c427d3c9834e9a1edb7beb4bf68d66a38d471b694e29f20f WatchSource:0}: Error finding container 050910dae45ef4e0c427d3c9834e9a1edb7beb4bf68d66a38d471b694e29f20f: Status 404 returned error can't find the container with id 050910dae45ef4e0c427d3c9834e9a1edb7beb4bf68d66a38d471b694e29f20f
Oct 30 03:27:47.248978 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:47.248937    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph" event={"ID":"3dfbdd36-8e2c-41d9-927b-510e1d9747af","Type":"ContainerStarted","Data":"0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928"}
Oct 30 03:27:47.249402 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:47.249003    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph" event={"ID":"3dfbdd36-8e2c-41d9-927b-510e1d9747af","Type":"ContainerStarted","Data":"050910dae45ef4e0c427d3c9834e9a1edb7beb4bf68d66a38d471b694e29f20f"}
Oct 30 03:27:47.291800 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:47.291204    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph" podStartSLOduration=2.291185909 podStartE2EDuration="2.291185909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:47.290669117 +0000 UTC m=+1931.496849364" watchObservedRunningTime="2026-10-30 03:27:47.291185909 +0000 UTC m=+1931.497366130"
Oct 30 03:27:47.330715 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:47.330668    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:27:48.046925 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:48.045998    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:54.083281 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:27:54.083231    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:27:56.046447 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:56.046411    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:56.059196 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:56.059168    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:56.302716 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:56.302618    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:27:57.330744 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:27:57.330701    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:28:07.340223 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:07.340186    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:28:07.348389 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:07.348352    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:28:14.105759 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:14.105724    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"]
Oct 30 03:28:24.081760 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:28:24.081719    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:28:29.130099 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:29.130049    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" containerID="cri-o://4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da" gracePeriod=45
Oct 30 03:28:29.341733 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:29.341691    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 03:28:29.878452 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:29.878424    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:28:30.010292 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.010194    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-tmp-dir\") pod \"aa138a32-4b5a-45a6-96c3-051dea2bd937\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") "
Oct 30 03:28:30.010292 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.010232    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-dshm\") pod \"aa138a32-4b5a-45a6-96c3-051dea2bd937\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") "
Oct 30 03:28:30.010292 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.010258    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-model-cache\") pod \"aa138a32-4b5a-45a6-96c3-051dea2bd937\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") "
Oct 30 03:28:30.010292 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.010296    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-home\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-home\") pod \"aa138a32-4b5a-45a6-96c3-051dea2bd937\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") "
Oct 30 03:28:30.010646 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.010334    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa138a32-4b5a-45a6-96c3-051dea2bd937-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa138a32-4b5a-45a6-96c3-051dea2bd937-cabundle-cert\") pod \"aa138a32-4b5a-45a6-96c3-051dea2bd937\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") "
Oct 30 03:28:30.010646 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.010360    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa138a32-4b5a-45a6-96c3-051dea2bd937-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa138a32-4b5a-45a6-96c3-051dea2bd937-tls-certs\") pod \"aa138a32-4b5a-45a6-96c3-051dea2bd937\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") "
Oct 30 03:28:30.010646 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.010439    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-kserve-provision-location\") pod \"aa138a32-4b5a-45a6-96c3-051dea2bd937\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") "
Oct 30 03:28:30.010646 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.010602    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-model-cache" pod "aa138a32-4b5a-45a6-96c3-051dea2bd937" (UID: "aa138a32-4b5a-45a6-96c3-051dea2bd937"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:30.010857 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.010680    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa138a32-4b5a-45a6-96c3-051dea2bd937-kube-api-access-xhxw2\" (UniqueName: \"kubernetes.io/projected/aa138a32-4b5a-45a6-96c3-051dea2bd937-kube-api-access-xhxw2\") pod \"aa138a32-4b5a-45a6-96c3-051dea2bd937\" (UID: \"aa138a32-4b5a-45a6-96c3-051dea2bd937\") "
Oct 30 03:28:30.010857 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.010771    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa138a32-4b5a-45a6-96c3-051dea2bd937-cabundle-cert" pod "aa138a32-4b5a-45a6-96c3-051dea2bd937" (UID: "aa138a32-4b5a-45a6-96c3-051dea2bd937"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:30.011114 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.011062    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.011114 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.011087    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa138a32-4b5a-45a6-96c3-051dea2bd937-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.011292 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.011155    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-home" pod "aa138a32-4b5a-45a6-96c3-051dea2bd937" (UID: "aa138a32-4b5a-45a6-96c3-051dea2bd937"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:30.012885 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.012860    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-dshm" pod "aa138a32-4b5a-45a6-96c3-051dea2bd937" (UID: "aa138a32-4b5a-45a6-96c3-051dea2bd937"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:30.013086 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.012890    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa138a32-4b5a-45a6-96c3-051dea2bd937-tls-certs" pod "aa138a32-4b5a-45a6-96c3-051dea2bd937" (UID: "aa138a32-4b5a-45a6-96c3-051dea2bd937"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:30.013149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.013126    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa138a32-4b5a-45a6-96c3-051dea2bd937-kube-api-access-xhxw2" pod "aa138a32-4b5a-45a6-96c3-051dea2bd937" (UID: "aa138a32-4b5a-45a6-96c3-051dea2bd937"). InnerVolumeSpecName "kube-api-access-xhxw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:30.022563 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.022532    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-tmp-dir" pod "aa138a32-4b5a-45a6-96c3-051dea2bd937" (UID: "aa138a32-4b5a-45a6-96c3-051dea2bd937"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:30.034202 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.034146    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-kserve-provision-location" pod "aa138a32-4b5a-45a6-96c3-051dea2bd937" (UID: "aa138a32-4b5a-45a6-96c3-051dea2bd937"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:30.112421 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.112385    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.112421 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.112417    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.112421 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.112427    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.112665 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.112436    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa138a32-4b5a-45a6-96c3-051dea2bd937-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.112665 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.112446    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa138a32-4b5a-45a6-96c3-051dea2bd937-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.112665 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.112454    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxw2\" (UniqueName: \"kubernetes.io/projected/aa138a32-4b5a-45a6-96c3-051dea2bd937-kube-api-access-xhxw2\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:28:30.456587 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.456549    4494 generic.go:356] "Generic (PLEG): container finished" podID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerID="4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da" exitCode=0
Oct 30 03:28:30.457054 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.456616    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"
Oct 30 03:28:30.457054 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.456637    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" event={"ID":"aa138a32-4b5a-45a6-96c3-051dea2bd937","Type":"ContainerDied","Data":"4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da"}
Oct 30 03:28:30.457054 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.456700    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb" event={"ID":"aa138a32-4b5a-45a6-96c3-051dea2bd937","Type":"ContainerDied","Data":"cf1747296a5544561bdb106802920afd5c8ce32e218a3febb820305b6f5b5dd9"}
Oct 30 03:28:30.457054 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.456723    4494 scope.go:122] "RemoveContainer" containerID="4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da"
Oct 30 03:28:30.466105 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.466088    4494 scope.go:122] "RemoveContainer" containerID="0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7"
Oct 30 03:28:30.474250 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.474232    4494 scope.go:122] "RemoveContainer" containerID="4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da"
Oct 30 03:28:30.474521 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:28:30.474502    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da\": container with ID starting with 4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da not found: ID does not exist" containerID="4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da"
Oct 30 03:28:30.474597 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.474535    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da"} err="failed to get container status \"4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da\": rpc error: code = NotFound desc = could not find container \"4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da\": container with ID starting with 4919a1b915310595f8e69fedc391daa8e9c24916cfa06c0b119a44f95538f4da not found: ID does not exist"
Oct 30 03:28:30.474597 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.474559    4494 scope.go:122] "RemoveContainer" containerID="0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7"
Oct 30 03:28:30.474834 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:28:30.474817    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7\": container with ID starting with 0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7 not found: ID does not exist" containerID="0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7"
Oct 30 03:28:30.474876 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.474840    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7"} err="failed to get container status \"0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7\": rpc error: code = NotFound desc = could not find container \"0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7\": container with ID starting with 0c627b3661e28cca597fabf47340c1f11190c8f4ee4a37666bd80db398f619b7 not found: ID does not exist"
Oct 30 03:28:30.526027 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.523668    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"]
Oct 30 03:28:30.534352 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:30.534312    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-65f985fb54-spkrb"]
Oct 30 03:28:31.927151 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:28:31.927117    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" path="/var/lib/kubelet/pods/aa138a32-4b5a-45a6-96c3-051dea2bd937/volumes"
Oct 30 03:28:54.084072 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:28:54.084025    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:29:06.709134 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.709091    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"]
Oct 30 03:29:06.709613 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.709459    4494 cpu_manager.go:414] "containerMap: removing container" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main"
Oct 30 03:29:06.709613 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.709471    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main"
Oct 30 03:29:06.709613 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.709490    4494 cpu_manager.go:414] "containerMap: removing container" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="storage-initializer"
Oct 30 03:29:06.709613 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.709496    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="storage-initializer"
Oct 30 03:29:06.709613 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.709578    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa138a32-4b5a-45a6-96c3-051dea2bd937" containerName="main"
Oct 30 03:29:06.712374 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.712359    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.726561 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.726523    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"]
Oct 30 03:29:06.728099 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.728070    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Oct 30 03:29:06.728860 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.728788    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Oct 30 03:29:06.736579 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.736554    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Oct 30 03:29:06.736579 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.736555    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Oct 30 03:29:06.736771 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.736581    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:29:06.828671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.828637    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-tvbss\""
Oct 30 03:29:06.835249 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.835210    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-model-cache\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.835491 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.835370    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-home\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.835491 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.835422    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.835491 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.835447    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.835723 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.835511    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tls-certs\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.835723 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.835616    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tmp-dir\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.835723 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.835652    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m4kf\" (UniqueName: \"kubernetes.io/projected/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kube-api-access-5m4kf\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.835723 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.835686    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-dshm\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937108 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937070    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-home\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937108 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937115    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937135    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937153    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tls-certs\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937172    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tmp-dir\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937197    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m4kf\" (UniqueName: \"kubernetes.io/projected/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kube-api-access-5m4kf\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937329    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-dshm\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937640 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937451    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-model-cache\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937640 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937569    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937640 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937614    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-home\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937798 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937676    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tmp-dir\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.937851 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.937821    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-model-cache\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.938063 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.938038    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.939930 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.939886    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-dshm\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.940090 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.940075    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tls-certs\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:06.963070 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:06.962983    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m4kf\" (UniqueName: \"kubernetes.io/projected/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kube-api-access-5m4kf\") pod \"auth-invalid-token-test-kserve-6bd9ccd4b-2czgc\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:07.030123 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:07.030080    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:07.348087 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:07.348058    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"]
Oct 30 03:29:07.348539 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:29:07.348516    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2dfaef5_f37a_49d5_83af_5c70bc5de3bf.slice/crio-6b784d694cf8cfaa93bfd4f013bcb4b6c344fa09af281fb062e4730f49aae165 WatchSource:0}: Error finding container 6b784d694cf8cfaa93bfd4f013bcb4b6c344fa09af281fb062e4730f49aae165: Status 404 returned error can't find the container with id 6b784d694cf8cfaa93bfd4f013bcb4b6c344fa09af281fb062e4730f49aae165
Oct 30 03:29:07.623321 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:07.623227    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" event={"ID":"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf","Type":"ContainerStarted","Data":"2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549"}
Oct 30 03:29:07.623321 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:07.623281    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" event={"ID":"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf","Type":"ContainerStarted","Data":"6b784d694cf8cfaa93bfd4f013bcb4b6c344fa09af281fb062e4730f49aae165"}
Oct 30 03:29:10.639438 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:10.639405    4494 generic.go:356] "Generic (PLEG): container finished" podID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerID="2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549" exitCode=0
Oct 30 03:29:10.639852 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:10.639450    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" event={"ID":"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf","Type":"ContainerDied","Data":"2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549"}
Oct 30 03:29:11.647054 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:11.647007    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" event={"ID":"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf","Type":"ContainerStarted","Data":"a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990"}
Oct 30 03:29:11.721817 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:11.721746    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podStartSLOduration=3.379063635 podStartE2EDuration="5.721727221s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.342663586s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:11.708288389 +0000 UTC m=+2015.914468633" watchObservedRunningTime="2026-10-30 03:29:11.721727221 +0000 UTC m=+2015.927907438"
Oct 30 03:29:12.030458 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:12.030428    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:17.030848 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:17.030809    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:29:17.032588 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:17.032561    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:29:23.229795 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.229763    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"]
Oct 30 03:29:23.240642 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:29:23.240594    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928" cmd=["/bin/sleep","15"]
Oct 30 03:29:23.240797 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.240660    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:29:23.240797 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:29:23.240679    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph" podUID="3dfbdd36-8e2c-41d9-927b-510e1d9747af" containerName="main" containerID="cri-o://0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928"
Oct 30 03:29:23.240797 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.240719    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph" podUID="3dfbdd36-8e2c-41d9-927b-510e1d9747af" containerName="main" containerID="cri-o://0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928" gracePeriod=60
Oct 30 03:29:23.292617 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.292572    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph" podUID="3dfbdd36-8e2c-41d9-927b-510e1d9747af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Oct 30 03:29:23.495733 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.495672    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:29:23.596524 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.596494    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-home\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-home\") pod \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") "
Oct 30 03:29:23.596671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.596545    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dfbdd36-8e2c-41d9-927b-510e1d9747af-kube-api-access-j2fln\" (UniqueName: \"kubernetes.io/projected/3dfbdd36-8e2c-41d9-927b-510e1d9747af-kube-api-access-j2fln\") pod \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") "
Oct 30 03:29:23.596671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.596568    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-model-cache\") pod \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") "
Oct 30 03:29:23.596671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.596620    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-dshm\") pod \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") "
Oct 30 03:29:23.596671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.596664    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tmp-dir\") pod \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") "
Oct 30 03:29:23.596855 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.596736    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tls-certs\" (UniqueName: \"kubernetes.io/secret/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tls-certs\") pod \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\" (UID: \"3dfbdd36-8e2c-41d9-927b-510e1d9747af\") "
Oct 30 03:29:23.596855 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.596831    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-home" pod "3dfbdd36-8e2c-41d9-927b-510e1d9747af" (UID: "3dfbdd36-8e2c-41d9-927b-510e1d9747af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.596855 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.596840    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-model-cache" pod "3dfbdd36-8e2c-41d9-927b-510e1d9747af" (UID: "3dfbdd36-8e2c-41d9-927b-510e1d9747af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.597071 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.597050    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.597187 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.597077    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.597187 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.597047    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tmp-dir" pod "3dfbdd36-8e2c-41d9-927b-510e1d9747af" (UID: "3dfbdd36-8e2c-41d9-927b-510e1d9747af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.598971 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.598947    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-dshm" pod "3dfbdd36-8e2c-41d9-927b-510e1d9747af" (UID: "3dfbdd36-8e2c-41d9-927b-510e1d9747af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:29:23.599081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.599061    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfbdd36-8e2c-41d9-927b-510e1d9747af-kube-api-access-j2fln" pod "3dfbdd36-8e2c-41d9-927b-510e1d9747af" (UID: "3dfbdd36-8e2c-41d9-927b-510e1d9747af"). InnerVolumeSpecName "kube-api-access-j2fln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:29:23.599081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.599064    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tls-certs" pod "3dfbdd36-8e2c-41d9-927b-510e1d9747af" (UID: "3dfbdd36-8e2c-41d9-927b-510e1d9747af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:29:23.698239 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.698216    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2fln\" (UniqueName: \"kubernetes.io/projected/3dfbdd36-8e2c-41d9-927b-510e1d9747af-kube-api-access-j2fln\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.698239 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.698235    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.698361 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.698247    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.698361 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.698256    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dfbdd36-8e2c-41d9-927b-510e1d9747af-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:29:23.718454 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.718431    4494 generic.go:356] "Generic (PLEG): container finished" podID="3dfbdd36-8e2c-41d9-927b-510e1d9747af" containerID="0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928" exitCode=0
Oct 30 03:29:23.718541 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.718479    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph" event={"ID":"3dfbdd36-8e2c-41d9-927b-510e1d9747af","Type":"ContainerDied","Data":"0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928"}
Oct 30 03:29:23.718541 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.718482    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"
Oct 30 03:29:23.718541 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.718520    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph" event={"ID":"3dfbdd36-8e2c-41d9-927b-510e1d9747af","Type":"ContainerDied","Data":"050910dae45ef4e0c427d3c9834e9a1edb7beb4bf68d66a38d471b694e29f20f"}
Oct 30 03:29:23.718541 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.718541    4494 scope.go:122] "RemoveContainer" containerID="0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928"
Oct 30 03:29:23.731699 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.731679    4494 scope.go:122] "RemoveContainer" containerID="0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928"
Oct 30 03:29:23.732021 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:29:23.732005    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928\": container with ID starting with 0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928 not found: ID does not exist" containerID="0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928"
Oct 30 03:29:23.732073 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.732028    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928"} err="failed to get container status \"0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928\": rpc error: code = NotFound desc = could not find container \"0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928\": container with ID starting with 0dd477fd509ca60cb53995da3465688f71a2f4074811348a11645a5555fd0928 not found: ID does not exist"
Oct 30 03:29:23.769183 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.769131    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"]
Oct 30 03:29:23.773233 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.773214    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-66d58bcdf9-fj9ph"]
Oct 30 03:29:23.928275 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:23.928231    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dfbdd36-8e2c-41d9-927b-510e1d9747af" path="/var/lib/kubelet/pods/3dfbdd36-8e2c-41d9-927b-510e1d9747af/volumes"
Oct 30 03:29:24.095472 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:29:24.095430    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:29:27.030758 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:27.030709    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:29:37.031538 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:37.031490    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:29:47.031180 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:47.031129    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:29:54.082097 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:29:54.082052    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:29:57.030687 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:29:57.030642    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:30:07.031607 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:30:07.031553    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:30:17.030698 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:30:17.030654    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:30:24.086251 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:30:24.086184    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:30:27.030774 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:30:27.030696    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:30:37.030715 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:30:37.030669    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:30:47.031481 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:30:47.031432    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:30:54.082829 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:30:54.082785    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:30:57.031575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:30:57.031528    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Oct 30 03:31:07.041043 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:07.040999    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:31:07.049176 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:07.049151    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:31:09.259714 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:09.259670    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"]
Oct 30 03:31:15.709229 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:15.709190    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"]
Oct 30 03:31:15.709712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:15.709411    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" podUID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" containerName="istio-proxy" containerID="cri-o://7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3" gracePeriod=30
Oct 30 03:31:20.959758 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:20.959723    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:31:21.004853 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.004763    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-podinfo\") pod \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") "
Oct 30 03:31:21.004853 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.004813    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-data\") pod \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") "
Oct 30 03:31:21.005100 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.004955    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-envoy\") pod \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") "
Oct 30 03:31:21.005100 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.005019    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-credential-socket\") pod \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") "
Oct 30 03:31:21.005100 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.005086    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-kube-api-access-w89kn\" (UniqueName: \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-kube-api-access-w89kn\") pod \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") "
Oct 30 03:31:21.005267 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.005129    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-certs\") pod \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") "
Oct 30 03:31:21.005267 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.005148    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-data" pod "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" (UID: "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.005267 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.005158    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-socket\") pod \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") "
Oct 30 03:31:21.005267 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.005210    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-token\") pod \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") "
Oct 30 03:31:21.005267 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.005244    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istiod-ca-cert\") pod \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\" (UID: \"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4\") "
Oct 30 03:31:21.005496 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.005304    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-credential-socket" pod "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" (UID: "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.005496 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.005399    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-certs" pod "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" (UID: "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.009944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.005384    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-socket" pod "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" (UID: "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.009944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.006200    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.009944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.006227    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.009944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.006234    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istiod-ca-cert" pod "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" (UID: "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:21.009944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.006243    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.009944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.007941    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-podinfo" pod "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" (UID: "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:31:21.015816 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.015718    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-kube-api-access-w89kn" pod "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" (UID: "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4"). InnerVolumeSpecName "kube-api-access-w89kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:21.017036 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.016893    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-envoy" pod "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" (UID: "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:21.017618 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.017593    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-token" pod "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" (UID: "4ad55ffb-b3f3-4d4c-946c-d13300b8dab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:21.107565 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.107513    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w89kn\" (UniqueName: \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-kube-api-access-w89kn\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.107565 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.107558    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.107565 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.107571    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.107565 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.107580    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.107565 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.107588    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.107883 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.107596    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:21.289158 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.289118    4494 generic.go:356] "Generic (PLEG): container finished" podID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" containerID="7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3" exitCode=0
Oct 30 03:31:21.289375 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.289187    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"
Oct 30 03:31:21.289375 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.289195    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" event={"ID":"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4","Type":"ContainerDied","Data":"7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3"}
Oct 30 03:31:21.289375 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.289300    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2" event={"ID":"4ad55ffb-b3f3-4d4c-946c-d13300b8dab4","Type":"ContainerDied","Data":"3b81502dfffe2bac39dc1e21ee573b2bab14d0d987424a0b63c113fea336620c"}
Oct 30 03:31:21.289375 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.289333    4494 scope.go:122] "RemoveContainer" containerID="7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3"
Oct 30 03:31:21.298951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.298926    4494 scope.go:122] "RemoveContainer" containerID="7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3"
Oct 30 03:31:21.299274 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:31:21.299253    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3\": container with ID starting with 7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3 not found: ID does not exist" containerID="7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3"
Oct 30 03:31:21.299327 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.299285    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3"} err="failed to get container status \"7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3\": rpc error: code = NotFound desc = could not find container \"7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3\": container with ID starting with 7ca8d3a8aac9df69ff0303fb6d07be0b65b8b19b3e0753d7fb0a44be5b2e2db3 not found: ID does not exist"
Oct 30 03:31:21.344009 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.343976    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"]
Oct 30 03:31:21.356415 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.356381    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-b5zb2"]
Oct 30 03:31:21.927027 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:21.926995    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" path="/var/lib/kubelet/pods/4ad55ffb-b3f3-4d4c-946c-d13300b8dab4/volumes"
Oct 30 03:31:24.082022 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:31:24.081986    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:31:24.283419 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:24.283370    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main" containerID="cri-o://a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990" gracePeriod=45
Oct 30 03:31:25.022277 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.022254    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:31:25.142710 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.142615    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kube-api-access-5m4kf\" (UniqueName: \"kubernetes.io/projected/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kube-api-access-5m4kf\") pod \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") "
Oct 30 03:31:25.142710 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.142669    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tmp-dir\") pod \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") "
Oct 30 03:31:25.142710 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.142712    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-home\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-home\") pod \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") "
Oct 30 03:31:25.143340 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.142800    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tls-certs\") pod \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") "
Oct 30 03:31:25.143340 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.142834    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-dshm\") pod \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") "
Oct 30 03:31:25.143340 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.142935    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-cabundle-cert\") pod \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") "
Oct 30 03:31:25.143340 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.142995    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-model-cache\") pod \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") "
Oct 30 03:31:25.143340 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.143105    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kserve-provision-location\") pod \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\" (UID: \"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf\") "
Oct 30 03:31:25.143668 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.143387    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-home" pod "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" (UID: "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:25.143668 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.143452    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.143864 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.143727    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-model-cache" pod "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" (UID: "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:25.143864 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.143795    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-cabundle-cert" pod "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" (UID: "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:31:25.145697 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.145662    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tls-certs" pod "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" (UID: "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:31:25.145697 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.145667    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kube-api-access-5m4kf" pod "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" (UID: "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf"). InnerVolumeSpecName "kube-api-access-5m4kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:31:25.145868 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.145691    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-dshm" pod "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" (UID: "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:25.159640 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.159617    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tmp-dir" pod "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" (UID: "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:25.171794 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.171769    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kserve-provision-location" pod "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" (UID: "e2dfaef5-f37a-49d5-83af-5c70bc5de3bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:31:25.244364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.244326    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.244364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.244365    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.244581 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.244376    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.244581 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.244388    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.244581 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.244399    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.244581 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.244410    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.244581 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.244421    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m4kf\" (UniqueName: \"kubernetes.io/projected/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf-kube-api-access-5m4kf\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:31:25.309832 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.309791    4494 generic.go:356] "Generic (PLEG): container finished" podID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerID="a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990" exitCode=0
Oct 30 03:31:25.310028 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.309857    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"
Oct 30 03:31:25.310028 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.309878    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" event={"ID":"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf","Type":"ContainerDied","Data":"a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990"}
Oct 30 03:31:25.310028 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.309971    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc" event={"ID":"e2dfaef5-f37a-49d5-83af-5c70bc5de3bf","Type":"ContainerDied","Data":"6b784d694cf8cfaa93bfd4f013bcb4b6c344fa09af281fb062e4730f49aae165"}
Oct 30 03:31:25.310028 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.309996    4494 scope.go:122] "RemoveContainer" containerID="a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990"
Oct 30 03:31:25.319172 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.319153    4494 scope.go:122] "RemoveContainer" containerID="2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549"
Oct 30 03:31:25.326949 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.326894    4494 scope.go:122] "RemoveContainer" containerID="a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990"
Oct 30 03:31:25.327198 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:31:25.327176    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990\": container with ID starting with a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990 not found: ID does not exist" containerID="a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990"
Oct 30 03:31:25.327251 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.327207    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990"} err="failed to get container status \"a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990\": rpc error: code = NotFound desc = could not find container \"a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990\": container with ID starting with a0acaeb40bf59f74dc572241dfe5a9aae787a361caa3ddbf1ed5b1e299f0a990 not found: ID does not exist"
Oct 30 03:31:25.327251 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.327223    4494 scope.go:122] "RemoveContainer" containerID="2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549"
Oct 30 03:31:25.327420 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:31:25.327406    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549\": container with ID starting with 2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549 not found: ID does not exist" containerID="2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549"
Oct 30 03:31:25.327463 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.327423    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549"} err="failed to get container status \"2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549\": rpc error: code = NotFound desc = could not find container \"2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549\": container with ID starting with 2eedb8a4bbdd742b05524ea222bd7e8eff29618730121ae304f5e68ef9e9e549 not found: ID does not exist"
Oct 30 03:31:25.361942 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.361915    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"]
Oct 30 03:31:25.374448 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.374423    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6bd9ccd4b-2czgc"]
Oct 30 03:31:25.932156 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:25.932121    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" path="/var/lib/kubelet/pods/e2dfaef5-f37a-49d5-83af-5c70bc5de3bf/volumes"
Oct 30 03:31:42.605863 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.605823    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"]
Oct 30 03:31:42.606309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606250    4494 cpu_manager.go:414] "containerMap: removing container" podUID="3dfbdd36-8e2c-41d9-927b-510e1d9747af" containerName="main"
Oct 30 03:31:42.606309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606263    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dfbdd36-8e2c-41d9-927b-510e1d9747af" containerName="main"
Oct 30 03:31:42.606309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606272    4494 cpu_manager.go:414] "containerMap: removing container" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="storage-initializer"
Oct 30 03:31:42.606309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606276    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="storage-initializer"
Oct 30 03:31:42.606309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606285    4494 cpu_manager.go:414] "containerMap: removing container" podUID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" containerName="istio-proxy"
Oct 30 03:31:42.606309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606290    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" containerName="istio-proxy"
Oct 30 03:31:42.606309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606297    4494 cpu_manager.go:414] "containerMap: removing container" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main"
Oct 30 03:31:42.606309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606301    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main"
Oct 30 03:31:42.606547 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606375    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dfbdd36-8e2c-41d9-927b-510e1d9747af" containerName="main"
Oct 30 03:31:42.606547 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606381    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad55ffb-b3f3-4d4c-946c-d13300b8dab4" containerName="istio-proxy"
Oct 30 03:31:42.606547 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.606386    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2dfaef5-f37a-49d5-83af-5c70bc5de3bf" containerName="main"
Oct 30 03:31:42.611145 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.611118    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.643531 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.643498    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Oct 30 03:31:42.643531 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.643522    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-rzfmv\""
Oct 30 03:31:42.643531 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.643515    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Oct 30 03:31:42.643855 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.643558    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Oct 30 03:31:42.645198 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.645170    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"]
Oct 30 03:31:42.699508 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.699458    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.699707 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.699527    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.699707 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.699633    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2474cf-d083-4315-b0df-e9ccda80834f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.699707 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.699670    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrwp\" (UniqueName: \"kubernetes.io/projected/ef2474cf-d083-4315-b0df-e9ccda80834f-kube-api-access-pkrwp\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.699707 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.699693    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.699868 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.699768    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.801120 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.801074    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.801310 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.801138    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2474cf-d083-4315-b0df-e9ccda80834f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.801310 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.801180    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrwp\" (UniqueName: \"kubernetes.io/projected/ef2474cf-d083-4315-b0df-e9ccda80834f-kube-api-access-pkrwp\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.801310 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.801206    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.801310 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.801226    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.801310 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.801288    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.801601 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.801573    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.801665 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.801593    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.801709 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.801666    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.803685 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.803652    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.803930 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.803914    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2474cf-d083-4315-b0df-e9ccda80834f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.825372 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.825337    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrwp\" (UniqueName: \"kubernetes.io/projected/ef2474cf-d083-4315-b0df-e9ccda80834f-kube-api-access-pkrwp\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:42.924413 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:42.924333    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:43.218844 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:43.218815    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"]
Oct 30 03:31:43.219009 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:31:43.218946    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2474cf_d083_4315_b0df_e9ccda80834f.slice/crio-23318ef003f242aa0cca77545296dcfc40769104ec2d3a87d81c33602effa955 WatchSource:0}: Error finding container 23318ef003f242aa0cca77545296dcfc40769104ec2d3a87d81c33602effa955: Status 404 returned error can't find the container with id 23318ef003f242aa0cca77545296dcfc40769104ec2d3a87d81c33602effa955
Oct 30 03:31:43.397226 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:43.397188    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8" event={"ID":"ef2474cf-d083-4315-b0df-e9ccda80834f","Type":"ContainerStarted","Data":"e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0"}
Oct 30 03:31:43.397415 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:43.397240    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8" event={"ID":"ef2474cf-d083-4315-b0df-e9ccda80834f","Type":"ContainerStarted","Data":"23318ef003f242aa0cca77545296dcfc40769104ec2d3a87d81c33602effa955"}
Oct 30 03:31:43.440558 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:43.440506    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8" podStartSLOduration=1.440489602 podStartE2EDuration="1.440489602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:31:43.439464577 +0000 UTC m=+2167.645644820" watchObservedRunningTime="2026-10-30 03:31:43.440489602 +0000 UTC m=+2167.646669825"
Oct 30 03:31:43.932341 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:43.932306    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:52.924566 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:52.924530    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:52.937159 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:52.937135    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:53.458026 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:31:53.457986    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:31:54.081686 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:31:54.081641    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:32:01.868089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:01.868048    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"]
Oct 30 03:32:01.871561 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:01.871542    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:01.887916 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:01.887879    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"]
Oct 30 03:32:01.898236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:01.898198    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Oct 30 03:32:01.898236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:01.898200    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Oct 30 03:32:01.898236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:01.898203    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:32:01.898236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:01.898211    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Oct 30 03:32:01.898728 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:01.898265    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-78jnx\""
Oct 30 03:32:01.898728 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:01.898344    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Oct 30 03:32:02.062401 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.062365    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-home\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.062575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.062426    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-model-cache\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.062575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.062530    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-kserve-provision-location\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.062575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.062573    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnlvl\" (UniqueName: \"kubernetes.io/projected/d1e959be-b639-46ca-92d1-414217bceb87-kube-api-access-lnlvl\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.062722 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.062671    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-dshm\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.062722 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.062689    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e959be-b639-46ca-92d1-414217bceb87-tls-certs\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.062834 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.062728    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-tmp-dir\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.062834 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.062751    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e959be-b639-46ca-92d1-414217bceb87-cabundle-cert\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164133 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164042    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-model-cache\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164133 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164088    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-kserve-provision-location\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164133 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164116    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnlvl\" (UniqueName: \"kubernetes.io/projected/d1e959be-b639-46ca-92d1-414217bceb87-kube-api-access-lnlvl\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164424 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164152    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-dshm\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164424 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164172    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e959be-b639-46ca-92d1-414217bceb87-tls-certs\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164424 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164211    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-tmp-dir\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164424 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164240    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e959be-b639-46ca-92d1-414217bceb87-cabundle-cert\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164424 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164293    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-home\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164678 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164554    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-model-cache\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164737 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164711    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-home\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.164797 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.164745    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-kserve-provision-location\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.165274 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.165181    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e959be-b639-46ca-92d1-414217bceb87-cabundle-cert\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.165490 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.165470    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-tmp-dir\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.166978 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.166944    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-dshm\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.167277 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.167259    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e959be-b639-46ca-92d1-414217bceb87-tls-certs\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.192889 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.192859    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnlvl\" (UniqueName: \"kubernetes.io/projected/d1e959be-b639-46ca-92d1-414217bceb87-kube-api-access-lnlvl\") pod \"auth-disabled-test-kserve-5f598d84c7-jgpfc\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.487488 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.487399    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:02.806928 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:32:02.806880    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e959be_b639_46ca_92d1_414217bceb87.slice/crio-ba3b659136f2e735fbcff07cf2d6e6fa8b969194612e6432db63b1eb4034bbaa WatchSource:0}: Error finding container ba3b659136f2e735fbcff07cf2d6e6fa8b969194612e6432db63b1eb4034bbaa: Status 404 returned error can't find the container with id ba3b659136f2e735fbcff07cf2d6e6fa8b969194612e6432db63b1eb4034bbaa
Oct 30 03:32:02.806928 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:02.806885    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"]
Oct 30 03:32:03.498122 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:03.498087    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" event={"ID":"d1e959be-b639-46ca-92d1-414217bceb87","Type":"ContainerStarted","Data":"a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15"}
Oct 30 03:32:03.498652 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:03.498152    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" event={"ID":"d1e959be-b639-46ca-92d1-414217bceb87","Type":"ContainerStarted","Data":"ba3b659136f2e735fbcff07cf2d6e6fa8b969194612e6432db63b1eb4034bbaa"}
Oct 30 03:32:05.509411 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:05.509319    4494 generic.go:356] "Generic (PLEG): container finished" podID="d1e959be-b639-46ca-92d1-414217bceb87" containerID="a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15" exitCode=0
Oct 30 03:32:05.509925 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:05.509404    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" event={"ID":"d1e959be-b639-46ca-92d1-414217bceb87","Type":"ContainerDied","Data":"a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15"}
Oct 30 03:32:06.516838 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:06.516794    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" event={"ID":"d1e959be-b639-46ca-92d1-414217bceb87","Type":"ContainerStarted","Data":"bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd"}
Oct 30 03:32:06.593380 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:06.593308    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podStartSLOduration=3.400378352 podStartE2EDuration="5.593292319s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.192913967s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:32:06.58023144 +0000 UTC m=+2190.786411688" watchObservedRunningTime="2026-10-30 03:32:06.593292319 +0000 UTC m=+2190.799472536"
Oct 30 03:32:07.487770 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:07.487728    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:12.488632 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:12.488587    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:32:12.490334 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:12.490300    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:32:18.043122 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.043021    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"]
Oct 30 03:32:18.054137 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:32:18.054096    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0" cmd=["/bin/sleep","15"]
Oct 30 03:32:18.054315 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.054163    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:32:18.054315 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:32:18.054185    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8" podUID="ef2474cf-d083-4315-b0df-e9ccda80834f" containerName="main" containerID="cri-o://e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0"
Oct 30 03:32:18.054315 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.054233    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8" podUID="ef2474cf-d083-4315-b0df-e9ccda80834f" containerName="main" containerID="cri-o://e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0" gracePeriod=60
Oct 30 03:32:18.411618 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.411591    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:32:18.510218 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510177    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-dshm\") pod \"ef2474cf-d083-4315-b0df-e9ccda80834f\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") "
Oct 30 03:32:18.510410 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510250    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-tmp-dir\") pod \"ef2474cf-d083-4315-b0df-e9ccda80834f\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") "
Oct 30 03:32:18.510410 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510289    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-home\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-home\") pod \"ef2474cf-d083-4315-b0df-e9ccda80834f\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") "
Oct 30 03:32:18.510410 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510346    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef2474cf-d083-4315-b0df-e9ccda80834f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2474cf-d083-4315-b0df-e9ccda80834f-tls-certs\") pod \"ef2474cf-d083-4315-b0df-e9ccda80834f\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") "
Oct 30 03:32:18.510410 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510380    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2474cf-d083-4315-b0df-e9ccda80834f-kube-api-access-pkrwp\" (UniqueName: \"kubernetes.io/projected/ef2474cf-d083-4315-b0df-e9ccda80834f-kube-api-access-pkrwp\") pod \"ef2474cf-d083-4315-b0df-e9ccda80834f\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") "
Oct 30 03:32:18.510645 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510448    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-model-cache\") pod \"ef2474cf-d083-4315-b0df-e9ccda80834f\" (UID: \"ef2474cf-d083-4315-b0df-e9ccda80834f\") "
Oct 30 03:32:18.510705 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510647    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-home" pod "ef2474cf-d083-4315-b0df-e9ccda80834f" (UID: "ef2474cf-d083-4315-b0df-e9ccda80834f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:18.510705 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510660    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-tmp-dir" pod "ef2474cf-d083-4315-b0df-e9ccda80834f" (UID: "ef2474cf-d083-4315-b0df-e9ccda80834f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:18.510834 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510815    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.510935 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510840    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.510935 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.510877    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-model-cache" pod "ef2474cf-d083-4315-b0df-e9ccda80834f" (UID: "ef2474cf-d083-4315-b0df-e9ccda80834f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:18.513399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.513293    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2474cf-d083-4315-b0df-e9ccda80834f-kube-api-access-pkrwp" pod "ef2474cf-d083-4315-b0df-e9ccda80834f" (UID: "ef2474cf-d083-4315-b0df-e9ccda80834f"). InnerVolumeSpecName "kube-api-access-pkrwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:18.513399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.513368    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef2474cf-d083-4315-b0df-e9ccda80834f-tls-certs" pod "ef2474cf-d083-4315-b0df-e9ccda80834f" (UID: "ef2474cf-d083-4315-b0df-e9ccda80834f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:18.513580 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.513406    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-dshm" pod "ef2474cf-d083-4315-b0df-e9ccda80834f" (UID: "ef2474cf-d083-4315-b0df-e9ccda80834f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:18.575548 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.575448    4494 generic.go:356] "Generic (PLEG): container finished" podID="ef2474cf-d083-4315-b0df-e9ccda80834f" containerID="e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0" exitCode=0
Oct 30 03:32:18.575548 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.575527    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"
Oct 30 03:32:18.575548 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.575531    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8" event={"ID":"ef2474cf-d083-4315-b0df-e9ccda80834f","Type":"ContainerDied","Data":"e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0"}
Oct 30 03:32:18.575833 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.575615    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8" event={"ID":"ef2474cf-d083-4315-b0df-e9ccda80834f","Type":"ContainerDied","Data":"23318ef003f242aa0cca77545296dcfc40769104ec2d3a87d81c33602effa955"}
Oct 30 03:32:18.575833 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.575641    4494 scope.go:122] "RemoveContainer" containerID="e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0"
Oct 30 03:32:18.586346 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.586317    4494 scope.go:122] "RemoveContainer" containerID="e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0"
Oct 30 03:32:18.586716 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:32:18.586695    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0\": container with ID starting with e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0 not found: ID does not exist" containerID="e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0"
Oct 30 03:32:18.586781 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.586725    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0"} err="failed to get container status \"e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0\": rpc error: code = NotFound desc = could not find container \"e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0\": container with ID starting with e183c1df3dc0ebf59464e84998ff3694dd84f80f15d8145b5d8453ff27ae99c0 not found: ID does not exist"
Oct 30 03:32:18.611480 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.611450    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.611480 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.611476    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2474cf-d083-4315-b0df-e9ccda80834f-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.611480 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.611487    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkrwp\" (UniqueName: \"kubernetes.io/projected/ef2474cf-d083-4315-b0df-e9ccda80834f-kube-api-access-pkrwp\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.611726 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.611496    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2474cf-d083-4315-b0df-e9ccda80834f-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:32:18.628196 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.628165    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"]
Oct 30 03:32:18.631178 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:18.631141    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-777624fk8"]
Oct 30 03:32:19.928061 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:19.928027    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2474cf-d083-4315-b0df-e9ccda80834f" path="/var/lib/kubelet/pods/ef2474cf-d083-4315-b0df-e9ccda80834f/volumes"
Oct 30 03:32:22.488317 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:22.488271    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:32:24.081643 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:32:24.081602    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:32:32.488442 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:32.488383    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:32:42.488490 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:42.488429    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:32:52.488185 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:32:52.488131    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:32:54.082792 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:32:54.082755    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:33:02.488131 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:02.488073    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:33:12.384041 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.383986    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"]
Oct 30 03:33:12.384703 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.384368    4494 cpu_manager.go:414] "containerMap: removing container" podUID="ef2474cf-d083-4315-b0df-e9ccda80834f" containerName="main"
Oct 30 03:33:12.384703 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.384379    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2474cf-d083-4315-b0df-e9ccda80834f" containerName="main"
Oct 30 03:33:12.384703 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.384468    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2474cf-d083-4315-b0df-e9ccda80834f" containerName="main"
Oct 30 03:33:12.386746 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.386725    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.402176 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.402146    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:33:12.402176 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.402167    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Oct 30 03:33:12.402176 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.402147    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Oct 30 03:33:12.402404 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.402202    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Oct 30 03:33:12.414446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.414420    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-q6h96\""
Oct 30 03:33:12.414639 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.414420    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Oct 30 03:33:12.416253 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.416229    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"]
Oct 30 03:33:12.488599 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.488560    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:33:12.515167 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.515123    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.515167 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.515173    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdkzz\" (UniqueName: \"kubernetes.io/projected/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kube-api-access-mdkzz\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.515438 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.515281    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tls-certs\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.515438 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.515336    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-home\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.515438 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.515418    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-dshm\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.515574 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.515446    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.515574 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.515464    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-model-cache\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.515574 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.515482    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.616391 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.616348    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-dshm\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.616391 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.616393    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.616672 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.616411    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-model-cache\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.616672 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.616544    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.616672 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.616592    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.616672 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.616622    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdkzz\" (UniqueName: \"kubernetes.io/projected/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kube-api-access-mdkzz\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.616882 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.616682    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tls-certs\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.616882 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.616725    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-home\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.616882 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.616802    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-model-cache\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.617087 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.616891    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.617145 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.617127    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.617226 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.617197    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-home\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.617226 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.617201    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.619246 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.619221    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-dshm\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.619446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.619431    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tls-certs\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.640566 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.640503    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdkzz\" (UniqueName: \"kubernetes.io/projected/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kube-api-access-mdkzz\") pod \"tokenizer-clean-path-test-kserve-846f48577-gmsns\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.700937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.700879    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:12.729692 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.727574    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"]
Oct 30 03:33:12.730513 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.730491    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.752158 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.752091    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"]
Oct 30 03:33:12.818556 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.818519    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/071face6-4616-498a-8f98-2ed795a18be5-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.818751 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.818725    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071face6-4616-498a-8f98-2ed795a18be5-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.818851 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.818833    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.818917 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.818876    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.920048 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.919945    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071face6-4616-498a-8f98-2ed795a18be5-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.920048 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.920001    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.920048 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.920021    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.920048 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.920041    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/071face6-4616-498a-8f98-2ed795a18be5-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.920472 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.920445    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.920546 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.920481    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.920790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.920768    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071face6-4616-498a-8f98-2ed795a18be5-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:12.922753 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:12.922733    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/071face6-4616-498a-8f98-2ed795a18be5-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:13.024771 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:13.024732    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"]
Oct 30 03:33:13.024999 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:33:13.024973    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83c2367_49ed_4fd9_878f_d13b3dd90bbb.slice/crio-01f1f4b8c45f2283a26c8eb007123f6f7c7caa9f70c4a40bcb0b6c2401c3b2a4 WatchSource:0}: Error finding container 01f1f4b8c45f2283a26c8eb007123f6f7c7caa9f70c4a40bcb0b6c2401c3b2a4: Status 404 returned error can't find the container with id 01f1f4b8c45f2283a26c8eb007123f6f7c7caa9f70c4a40bcb0b6c2401c3b2a4
Oct 30 03:33:13.048671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:13.048648    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:13.348704 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:13.348671    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"]
Oct 30 03:33:13.351009 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:33:13.350971    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071face6_4616_498a_8f98_2ed795a18be5.slice/crio-57cfcb58bb9080b75a394798a5a8adfe77f69b64c39a3c46057e5ecde8d54dd9 WatchSource:0}: Error finding container 57cfcb58bb9080b75a394798a5a8adfe77f69b64c39a3c46057e5ecde8d54dd9: Status 404 returned error can't find the container with id 57cfcb58bb9080b75a394798a5a8adfe77f69b64c39a3c46057e5ecde8d54dd9
Oct 30 03:33:13.847772 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:13.847733    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d" event={"ID":"071face6-4616-498a-8f98-2ed795a18be5","Type":"ContainerStarted","Data":"d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504"}
Oct 30 03:33:13.848258 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:13.847826    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d" event={"ID":"071face6-4616-498a-8f98-2ed795a18be5","Type":"ContainerStarted","Data":"57cfcb58bb9080b75a394798a5a8adfe77f69b64c39a3c46057e5ecde8d54dd9"}
Oct 30 03:33:13.850021 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:13.849991    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns" event={"ID":"a83c2367-49ed-4fd9-878f-d13b3dd90bbb","Type":"ContainerStarted","Data":"0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95"}
Oct 30 03:33:13.850177 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:13.850050    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns" event={"ID":"a83c2367-49ed-4fd9-878f-d13b3dd90bbb","Type":"ContainerStarted","Data":"01f1f4b8c45f2283a26c8eb007123f6f7c7caa9f70c4a40bcb0b6c2401c3b2a4"}
Oct 30 03:33:15.863326 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:15.863289    4494 generic.go:356] "Generic (PLEG): container finished" podID="a83c2367-49ed-4fd9-878f-d13b3dd90bbb" containerID="0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95" exitCode=0
Oct 30 03:33:15.863829 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:15.863360    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns" event={"ID":"a83c2367-49ed-4fd9-878f-d13b3dd90bbb","Type":"ContainerDied","Data":"0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95"}
Oct 30 03:33:16.873941 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:16.873875    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns" event={"ID":"a83c2367-49ed-4fd9-878f-d13b3dd90bbb","Type":"ContainerStarted","Data":"d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8"}
Oct 30 03:33:16.875470 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:16.875438    4494 generic.go:356] "Generic (PLEG): container finished" podID="071face6-4616-498a-8f98-2ed795a18be5" containerID="d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504" exitCode=0
Oct 30 03:33:16.875594 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:16.875488    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d" event={"ID":"071face6-4616-498a-8f98-2ed795a18be5","Type":"ContainerDied","Data":"d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504"}
Oct 30 03:33:16.932700 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:16.932640    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns" podStartSLOduration=2.386808682 podStartE2EDuration="4.932620063s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.545811381s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:16.931452171 +0000 UTC m=+2261.137632390" watchObservedRunningTime="2026-10-30 03:33:16.932620063 +0000 UTC m=+2261.138800282"
Oct 30 03:33:17.701235 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:17.701196    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:17.882519 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:17.882481    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d" event={"ID":"071face6-4616-498a-8f98-2ed795a18be5","Type":"ContainerStarted","Data":"6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4"}
Oct 30 03:33:17.927701 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:17.927647    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d" podStartSLOduration=3.392534248 podStartE2EDuration="5.927632109s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.535097861s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:33:17.926501873 +0000 UTC m=+2262.132682114" watchObservedRunningTime="2026-10-30 03:33:17.927632109 +0000 UTC m=+2262.133812371"
Oct 30 03:33:22.488204 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:22.488153    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:33:22.702051 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:22.702010    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:22.717714 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:22.717679    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:22.922087 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:22.922055    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:33:24.084642 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:33:24.084590    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:33:27.883375 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:27.883325    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:33:32.488717 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:32.488671    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:33:42.488545 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:42.488485    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:33:47.886508 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:47.886401    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d" podUID="071face6-4616-498a-8f98-2ed795a18be5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 30 03:33:52.488668 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:52.488612    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:33:54.082998 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:33:54.082951    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:33:57.905292 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:33:57.905253    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:34:00.006998 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.006965    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d_071face6-4616-498a-8f98-2ed795a18be5/main/0.log"
Oct 30 03:34:00.433150 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.433110    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"]
Oct 30 03:34:00.445568 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:00.445499    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8" cmd=["/bin/sleep","15"]
Oct 30 03:34:00.445740 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.445587    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:34:00.445740 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:00.445603    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns" podUID="a83c2367-49ed-4fd9-878f-d13b3dd90bbb" containerName="main" containerID="cri-o://d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8"
Oct 30 03:34:00.445740 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.445665    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns" podUID="a83c2367-49ed-4fd9-878f-d13b3dd90bbb" containerName="main" containerID="cri-o://d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8" gracePeriod=60
Oct 30 03:34:00.450362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.450336    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"]
Oct 30 03:34:00.450601 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.450554    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d" podUID="071face6-4616-498a-8f98-2ed795a18be5" containerName="main" containerID="cri-o://6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4" gracePeriod=30
Oct 30 03:34:00.812063 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.812033    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:34:00.989370 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.989328    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kube-api-access-mdkzz\" (UniqueName: \"kubernetes.io/projected/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kube-api-access-mdkzz\") pod \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") "
Oct 30 03:34:00.989536 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.989393    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kserve-provision-location\") pod \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") "
Oct 30 03:34:00.989588 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.989555    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tmp-dir\") pod \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") "
Oct 30 03:34:00.989651 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.989635    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-dshm\") pod \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") "
Oct 30 03:34:00.989708 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.989696    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-home\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-home\") pod \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") "
Oct 30 03:34:00.989754 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.989729    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-model-cache\") pod \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") "
Oct 30 03:34:00.989809 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.989788    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-cabundle-cert\") pod \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") "
Oct 30 03:34:00.989851 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.989824    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tls-certs\") pod \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\" (UID: \"a83c2367-49ed-4fd9-878f-d13b3dd90bbb\") "
Oct 30 03:34:00.989962 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.989932    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tmp-dir" pod "a83c2367-49ed-4fd9-878f-d13b3dd90bbb" (UID: "a83c2367-49ed-4fd9-878f-d13b3dd90bbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:00.990221 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.990185    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:00.990567 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.990402    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-home" pod "a83c2367-49ed-4fd9-878f-d13b3dd90bbb" (UID: "a83c2367-49ed-4fd9-878f-d13b3dd90bbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:00.990994 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.990966    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-cabundle-cert" pod "a83c2367-49ed-4fd9-878f-d13b3dd90bbb" (UID: "a83c2367-49ed-4fd9-878f-d13b3dd90bbb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:00.991121 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.991040    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-model-cache" pod "a83c2367-49ed-4fd9-878f-d13b3dd90bbb" (UID: "a83c2367-49ed-4fd9-878f-d13b3dd90bbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:00.992096 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.992071    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kube-api-access-mdkzz" pod "a83c2367-49ed-4fd9-878f-d13b3dd90bbb" (UID: "a83c2367-49ed-4fd9-878f-d13b3dd90bbb"). InnerVolumeSpecName "kube-api-access-mdkzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:00.992203 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.992188    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tls-certs" pod "a83c2367-49ed-4fd9-878f-d13b3dd90bbb" (UID: "a83c2367-49ed-4fd9-878f-d13b3dd90bbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:00.992603 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:00.992588    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-dshm" pod "a83c2367-49ed-4fd9-878f-d13b3dd90bbb" (UID: "a83c2367-49ed-4fd9-878f-d13b3dd90bbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:01.010133 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.010079    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kserve-provision-location" pod "a83c2367-49ed-4fd9-878f-d13b3dd90bbb" (UID: "a83c2367-49ed-4fd9-878f-d13b3dd90bbb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:01.090723 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.090691    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdkzz\" (UniqueName: \"kubernetes.io/projected/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kube-api-access-mdkzz\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.090723 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.090721    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.090723 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.090731    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.091149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.090739    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.091149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.090747    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.091149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.090755    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.091149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.090764    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a83c2367-49ed-4fd9-878f-d13b3dd90bbb-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.094148 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.094124    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:34:01.136601 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.136564    4494 generic.go:356] "Generic (PLEG): container finished" podID="071face6-4616-498a-8f98-2ed795a18be5" containerID="6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4" exitCode=0
Oct 30 03:34:01.137689 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.136622    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"
Oct 30 03:34:01.137689 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.136653    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d" event={"ID":"071face6-4616-498a-8f98-2ed795a18be5","Type":"ContainerDied","Data":"6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4"}
Oct 30 03:34:01.137689 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.136740    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d" event={"ID":"071face6-4616-498a-8f98-2ed795a18be5","Type":"ContainerDied","Data":"57cfcb58bb9080b75a394798a5a8adfe77f69b64c39a3c46057e5ecde8d54dd9"}
Oct 30 03:34:01.137689 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.137363    4494 scope.go:122] "RemoveContainer" containerID="6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4"
Oct 30 03:34:01.140887 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.140859    4494 generic.go:356] "Generic (PLEG): container finished" podID="a83c2367-49ed-4fd9-878f-d13b3dd90bbb" containerID="d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8" exitCode=0
Oct 30 03:34:01.141049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.140963    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"
Oct 30 03:34:01.141049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.140958    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns" event={"ID":"a83c2367-49ed-4fd9-878f-d13b3dd90bbb","Type":"ContainerDied","Data":"d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8"}
Oct 30 03:34:01.141049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.141042    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns" event={"ID":"a83c2367-49ed-4fd9-878f-d13b3dd90bbb","Type":"ContainerDied","Data":"01f1f4b8c45f2283a26c8eb007123f6f7c7caa9f70c4a40bcb0b6c2401c3b2a4"}
Oct 30 03:34:01.149647 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.149543    4494 scope.go:122] "RemoveContainer" containerID="d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504"
Oct 30 03:34:01.161817 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.161796    4494 scope.go:122] "RemoveContainer" containerID="6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4"
Oct 30 03:34:01.162208 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:01.162186    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4\": container with ID starting with 6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4 not found: ID does not exist" containerID="6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4"
Oct 30 03:34:01.162313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.162223    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4"} err="failed to get container status \"6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4\": rpc error: code = NotFound desc = could not find container \"6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4\": container with ID starting with 6a866133ca3944c3efd59d62e8f9e2eacfd59c66d7517e834b8b1bbbe0de0fc4 not found: ID does not exist"
Oct 30 03:34:01.162313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.162249    4494 scope.go:122] "RemoveContainer" containerID="d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504"
Oct 30 03:34:01.162578 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:01.162546    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504\": container with ID starting with d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504 not found: ID does not exist" containerID="d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504"
Oct 30 03:34:01.162634 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.162587    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504"} err="failed to get container status \"d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504\": rpc error: code = NotFound desc = could not find container \"d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504\": container with ID starting with d2aa485d640cf1fd85171359e8ac2d355fe8c5a93bcaef90ab72fcd7a1f42504 not found: ID does not exist"
Oct 30 03:34:01.162634 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.162605    4494 scope.go:122] "RemoveContainer" containerID="d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8"
Oct 30 03:34:01.192348 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.192293    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"]
Oct 30 03:34:01.192551 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.192460    4494 scope.go:122] "RemoveContainer" containerID="0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95"
Oct 30 03:34:01.203089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.197183    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-tmp-dir\") pod \"071face6-4616-498a-8f98-2ed795a18be5\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") "
Oct 30 03:34:01.203089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.197248    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/071face6-4616-498a-8f98-2ed795a18be5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071face6-4616-498a-8f98-2ed795a18be5-cabundle-cert\") pod \"071face6-4616-498a-8f98-2ed795a18be5\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") "
Oct 30 03:34:01.203089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.197298    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/071face6-4616-498a-8f98-2ed795a18be5-tls-certs\" (UniqueName: \"kubernetes.io/secret/071face6-4616-498a-8f98-2ed795a18be5-tls-certs\") pod \"071face6-4616-498a-8f98-2ed795a18be5\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") "
Oct 30 03:34:01.203089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.197377    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-kserve-provision-location\") pod \"071face6-4616-498a-8f98-2ed795a18be5\" (UID: \"071face6-4616-498a-8f98-2ed795a18be5\") "
Oct 30 03:34:01.203089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.198326    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071face6-4616-498a-8f98-2ed795a18be5-cabundle-cert" pod "071face6-4616-498a-8f98-2ed795a18be5" (UID: "071face6-4616-498a-8f98-2ed795a18be5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:01.203089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.198758    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-tmp-dir" pod "071face6-4616-498a-8f98-2ed795a18be5" (UID: "071face6-4616-498a-8f98-2ed795a18be5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:01.203089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.201732    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/071face6-4616-498a-8f98-2ed795a18be5-tls-certs" pod "071face6-4616-498a-8f98-2ed795a18be5" (UID: "071face6-4616-498a-8f98-2ed795a18be5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:01.206498 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.206458    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-846f48577-gmsns"]
Oct 30 03:34:01.226208 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.226051    4494 scope.go:122] "RemoveContainer" containerID="d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8"
Oct 30 03:34:01.226582 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:01.226546    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8\": container with ID starting with d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8 not found: ID does not exist" containerID="d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8"
Oct 30 03:34:01.226775 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.226591    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8"} err="failed to get container status \"d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8\": rpc error: code = NotFound desc = could not find container \"d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8\": container with ID starting with d9ba2085ecc7d54e00b73909844535d6c62e2b162e33d6467442f900a2ca36c8 not found: ID does not exist"
Oct 30 03:34:01.226775 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.226615    4494 scope.go:122] "RemoveContainer" containerID="0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95"
Oct 30 03:34:01.226960 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:01.226929    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95\": container with ID starting with 0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95 not found: ID does not exist" containerID="0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95"
Oct 30 03:34:01.227061 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.226963    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95"} err="failed to get container status \"0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95\": rpc error: code = NotFound desc = could not find container \"0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95\": container with ID starting with 0fdaac96741de896bfcb94d8e04a112762a522b8d2e7a8622a32af885b002a95 not found: ID does not exist"
Oct 30 03:34:01.232500 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.232462    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-kserve-provision-location" pod "071face6-4616-498a-8f98-2ed795a18be5" (UID: "071face6-4616-498a-8f98-2ed795a18be5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:01.298593 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.298550    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.298593 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.298585    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/071face6-4616-498a-8f98-2ed795a18be5-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.298593 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.298597    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/071face6-4616-498a-8f98-2ed795a18be5-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.298593 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.298605    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/071face6-4616-498a-8f98-2ed795a18be5-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:01.482575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.482490    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"]
Oct 30 03:34:01.486813 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.486786    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6f446c796b-gmw9d"]
Oct 30 03:34:01.927290 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.927243    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="071face6-4616-498a-8f98-2ed795a18be5" path="/var/lib/kubelet/pods/071face6-4616-498a-8f98-2ed795a18be5/volumes"
Oct 30 03:34:01.927816 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:01.927794    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83c2367-49ed-4fd9-878f-d13b3dd90bbb" path="/var/lib/kubelet/pods/a83c2367-49ed-4fd9-878f-d13b3dd90bbb/volumes"
Oct 30 03:34:02.499059 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:02.499023    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:34:02.507017 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:02.506981    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:34:08.025770 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:08.025732    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"]
Oct 30 03:34:23.049353 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.049282    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" containerID="cri-o://bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd" gracePeriod=45
Oct 30 03:34:23.500180 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.500079    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 30 03:34:23.804756 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.804727    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:34:23.895403 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.895368    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-dshm\") pod \"d1e959be-b639-46ca-92d1-414217bceb87\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") "
Oct 30 03:34:23.895627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.895432    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1e959be-b639-46ca-92d1-414217bceb87-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e959be-b639-46ca-92d1-414217bceb87-tls-certs\") pod \"d1e959be-b639-46ca-92d1-414217bceb87\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") "
Oct 30 03:34:23.895627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.895489    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-kserve-provision-location\") pod \"d1e959be-b639-46ca-92d1-414217bceb87\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") "
Oct 30 03:34:23.895627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.895524    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-model-cache\") pod \"d1e959be-b639-46ca-92d1-414217bceb87\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") "
Oct 30 03:34:23.895627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.895592    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-tmp-dir\") pod \"d1e959be-b639-46ca-92d1-414217bceb87\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") "
Oct 30 03:34:23.895855 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.895644    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-home\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-home\") pod \"d1e959be-b639-46ca-92d1-414217bceb87\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") "
Oct 30 03:34:23.895855 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.895673    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e959be-b639-46ca-92d1-414217bceb87-kube-api-access-lnlvl\" (UniqueName: \"kubernetes.io/projected/d1e959be-b639-46ca-92d1-414217bceb87-kube-api-access-lnlvl\") pod \"d1e959be-b639-46ca-92d1-414217bceb87\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") "
Oct 30 03:34:23.895855 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.895712    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e959be-b639-46ca-92d1-414217bceb87-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e959be-b639-46ca-92d1-414217bceb87-cabundle-cert\") pod \"d1e959be-b639-46ca-92d1-414217bceb87\" (UID: \"d1e959be-b639-46ca-92d1-414217bceb87\") "
Oct 30 03:34:23.896933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.896129    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-model-cache" pod "d1e959be-b639-46ca-92d1-414217bceb87" (UID: "d1e959be-b639-46ca-92d1-414217bceb87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:23.896933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.896511    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e959be-b639-46ca-92d1-414217bceb87-cabundle-cert" pod "d1e959be-b639-46ca-92d1-414217bceb87" (UID: "d1e959be-b639-46ca-92d1-414217bceb87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:23.896933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.896658    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-home" pod "d1e959be-b639-46ca-92d1-414217bceb87" (UID: "d1e959be-b639-46ca-92d1-414217bceb87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:23.898001 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.897965    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-dshm" pod "d1e959be-b639-46ca-92d1-414217bceb87" (UID: "d1e959be-b639-46ca-92d1-414217bceb87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:23.898191 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.898165    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e959be-b639-46ca-92d1-414217bceb87-tls-certs" pod "d1e959be-b639-46ca-92d1-414217bceb87" (UID: "d1e959be-b639-46ca-92d1-414217bceb87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:23.898489 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.898458    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e959be-b639-46ca-92d1-414217bceb87-kube-api-access-lnlvl" pod "d1e959be-b639-46ca-92d1-414217bceb87" (UID: "d1e959be-b639-46ca-92d1-414217bceb87"). InnerVolumeSpecName "kube-api-access-lnlvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:23.907981 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.907939    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-tmp-dir" pod "d1e959be-b639-46ca-92d1-414217bceb87" (UID: "d1e959be-b639-46ca-92d1-414217bceb87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:23.920646 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.920602    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-kserve-provision-location" pod "d1e959be-b639-46ca-92d1-414217bceb87" (UID: "d1e959be-b639-46ca-92d1-414217bceb87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:23.996704 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.996666    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e959be-b639-46ca-92d1-414217bceb87-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:23.996704 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.996696    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:23.996704 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.996705    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e959be-b639-46ca-92d1-414217bceb87-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:23.996704 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.996713    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:23.997015 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.996724    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:23.997015 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.996754    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:23.997015 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.996762    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1e959be-b639-46ca-92d1-414217bceb87-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:23.997015 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:23.996769    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnlvl\" (UniqueName: \"kubernetes.io/projected/d1e959be-b639-46ca-92d1-414217bceb87-kube-api-access-lnlvl\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:24.082196 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:24.082107    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:34:24.252598 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.252564    4494 generic.go:356] "Generic (PLEG): container finished" podID="d1e959be-b639-46ca-92d1-414217bceb87" containerID="bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd" exitCode=0
Oct 30 03:34:24.252766 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.252620    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"
Oct 30 03:34:24.252766 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.252641    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" event={"ID":"d1e959be-b639-46ca-92d1-414217bceb87","Type":"ContainerDied","Data":"bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd"}
Oct 30 03:34:24.252766 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.252696    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc" event={"ID":"d1e959be-b639-46ca-92d1-414217bceb87","Type":"ContainerDied","Data":"ba3b659136f2e735fbcff07cf2d6e6fa8b969194612e6432db63b1eb4034bbaa"}
Oct 30 03:34:24.252766 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.252719    4494 scope.go:122] "RemoveContainer" containerID="bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd"
Oct 30 03:34:24.261508 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.261490    4494 scope.go:122] "RemoveContainer" containerID="a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15"
Oct 30 03:34:24.269886 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.269867    4494 scope.go:122] "RemoveContainer" containerID="bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd"
Oct 30 03:34:24.270187 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:24.270167    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd\": container with ID starting with bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd not found: ID does not exist" containerID="bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd"
Oct 30 03:34:24.270236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.270200    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd"} err="failed to get container status \"bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd\": rpc error: code = NotFound desc = could not find container \"bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd\": container with ID starting with bbfdf9c1a3b9b40f235e219fae6e4f47d982e442f4d4c9df9615d6f595d90ffd not found: ID does not exist"
Oct 30 03:34:24.270236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.270218    4494 scope.go:122] "RemoveContainer" containerID="a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15"
Oct 30 03:34:24.270441 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:24.270423    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15\": container with ID starting with a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15 not found: ID does not exist" containerID="a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15"
Oct 30 03:34:24.270481 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.270448    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15"} err="failed to get container status \"a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15\": rpc error: code = NotFound desc = could not find container \"a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15\": container with ID starting with a205fbc04ec80fb1ab590ce0bc42310cc3b53f63995740fbd3f96fc555223b15 not found: ID does not exist"
Oct 30 03:34:24.304516 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.304484    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"]
Oct 30 03:34:24.317001 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:24.316964    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5f598d84c7-jgpfc"]
Oct 30 03:34:25.926517 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:25.926476    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e959be-b639-46ca-92d1-414217bceb87" path="/var/lib/kubelet/pods/d1e959be-b639-46ca-92d1-414217bceb87/volumes"
Oct 30 03:34:31.260261 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:31.260221    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"]
Oct 30 03:34:31.260727 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:31.260450    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy" containerID="cri-o://27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db" gracePeriod=30
Oct 30 03:34:31.801543 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:31.801502    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:34:31.801755 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:31.801587    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:34:36.521565 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.521529    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:34:36.612653 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.612607    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-podinfo\") pod \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") "
Oct 30 03:34:36.612865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.612676    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-kube-api-access-shwbq\" (UniqueName: \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-kube-api-access-shwbq\") pod \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") "
Oct 30 03:34:36.612865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.612714    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-credential-socket\") pod \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") "
Oct 30 03:34:36.612865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.612735    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istiod-ca-cert\") pod \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") "
Oct 30 03:34:36.612865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.612773    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-socket\") pod \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") "
Oct 30 03:34:36.612865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.612834    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-envoy\") pod \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") "
Oct 30 03:34:36.612865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.612861    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-certs\") pod \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") "
Oct 30 03:34:36.613203 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.612934    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-token\" (UniqueName: \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-token\") pod \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") "
Oct 30 03:34:36.613203 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.612974    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-data\") pod \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\" (UID: \"92a3121c-9c1e-4085-a4b6-151ccd0438e5\") "
Oct 30 03:34:36.613203 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.613075    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-credential-socket" pod "92a3121c-9c1e-4085-a4b6-151ccd0438e5" (UID: "92a3121c-9c1e-4085-a4b6-151ccd0438e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:36.613203 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.613141    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-socket" pod "92a3121c-9c1e-4085-a4b6-151ccd0438e5" (UID: "92a3121c-9c1e-4085-a4b6-151ccd0438e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:36.613395 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.613295    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-certs" pod "92a3121c-9c1e-4085-a4b6-151ccd0438e5" (UID: "92a3121c-9c1e-4085-a4b6-151ccd0438e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:36.613395 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.613307    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istiod-ca-cert" pod "92a3121c-9c1e-4085-a4b6-151ccd0438e5" (UID: "92a3121c-9c1e-4085-a4b6-151ccd0438e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:36.613500 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.613462    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-data" pod "92a3121c-9c1e-4085-a4b6-151ccd0438e5" (UID: "92a3121c-9c1e-4085-a4b6-151ccd0438e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:36.613584 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.613559    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:36.613710 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.613587    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:36.613710 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.613601    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:36.613710 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.613616    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:36.613710 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.613655    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:36.615392 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.615360    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-podinfo" pod "92a3121c-9c1e-4085-a4b6-151ccd0438e5" (UID: "92a3121c-9c1e-4085-a4b6-151ccd0438e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:34:36.615583 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.615565    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-token" pod "92a3121c-9c1e-4085-a4b6-151ccd0438e5" (UID: "92a3121c-9c1e-4085-a4b6-151ccd0438e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:36.615792 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.615770    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-kube-api-access-shwbq" pod "92a3121c-9c1e-4085-a4b6-151ccd0438e5" (UID: "92a3121c-9c1e-4085-a4b6-151ccd0438e5"). InnerVolumeSpecName "kube-api-access-shwbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:36.615792 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.615780    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-envoy" pod "92a3121c-9c1e-4085-a4b6-151ccd0438e5" (UID: "92a3121c-9c1e-4085-a4b6-151ccd0438e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:36.714459 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.714423    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:36.714459 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.714456    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:36.714459 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.714464    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a3121c-9c1e-4085-a4b6-151ccd0438e5-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:36.714726 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:36.714473    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shwbq\" (UniqueName: \"kubernetes.io/projected/92a3121c-9c1e-4085-a4b6-151ccd0438e5-kube-api-access-shwbq\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:34:37.313467 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.313431    4494 generic.go:356] "Generic (PLEG): container finished" podID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerID="27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db" exitCode=0
Oct 30 03:34:37.313620 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.313484    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"
Oct 30 03:34:37.313620 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.313482    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" event={"ID":"92a3121c-9c1e-4085-a4b6-151ccd0438e5","Type":"ContainerDied","Data":"27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db"}
Oct 30 03:34:37.313620 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.313613    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg" event={"ID":"92a3121c-9c1e-4085-a4b6-151ccd0438e5","Type":"ContainerDied","Data":"225c885bbaf76cb38d40a32c7dab19759b7a8ee2c38c56ca3e6433d7f51f859f"}
Oct 30 03:34:37.313742 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.313641    4494 scope.go:122] "RemoveContainer" containerID="27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db"
Oct 30 03:34:37.322530 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.322508    4494 scope.go:122] "RemoveContainer" containerID="951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f"
Oct 30 03:34:37.331438 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.331418    4494 scope.go:122] "RemoveContainer" containerID="27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db"
Oct 30 03:34:37.331775 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:37.331754    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db\": container with ID starting with 27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db not found: ID does not exist" containerID="27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db"
Oct 30 03:34:37.331834 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.331785    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db"} err="failed to get container status \"27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db\": rpc error: code = NotFound desc = could not find container \"27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db\": container with ID starting with 27e19390c3df130c663b78e3b25bd62c91cad2e7a4e016e14cbdf152d56d50db not found: ID does not exist"
Oct 30 03:34:37.331834 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.331804    4494 scope.go:122] "RemoveContainer" containerID="951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f"
Oct 30 03:34:37.332120 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:37.332102    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f\": container with ID starting with 951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f not found: ID does not exist" containerID="951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f"
Oct 30 03:34:37.332175 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.332132    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f"} err="failed to get container status \"951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f\": rpc error: code = NotFound desc = could not find container \"951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f\": container with ID starting with 951f0944ea81a1053f70fd6c7ea39d4fe799e79a70d7670f0daf38183b56f19f not found: ID does not exist"
Oct 30 03:34:37.377540 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.377494    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"]
Oct 30 03:34:37.395253 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.392834    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-t5mxg"]
Oct 30 03:34:37.926890 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:37.926858    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" path="/var/lib/kubelet/pods/92a3121c-9c1e-4085-a4b6-151ccd0438e5/volumes"
Oct 30 03:34:54.082149 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:34:54.082114    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:34:55.683190 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683134    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"]
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683484    4494 cpu_manager.go:414] "containerMap: removing container" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683495    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683508    4494 cpu_manager.go:414] "containerMap: removing container" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683514    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683528    4494 cpu_manager.go:414] "containerMap: removing container" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683533    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683540    4494 cpu_manager.go:414] "containerMap: removing container" podUID="a83c2367-49ed-4fd9-878f-d13b3dd90bbb" containerName="main"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683544    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83c2367-49ed-4fd9-878f-d13b3dd90bbb" containerName="main"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683551    4494 cpu_manager.go:414] "containerMap: removing container" podUID="071face6-4616-498a-8f98-2ed795a18be5" containerName="main"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683556    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071face6-4616-498a-8f98-2ed795a18be5" containerName="main"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683566    4494 cpu_manager.go:414] "containerMap: removing container" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="storage-initializer"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683570    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="storage-initializer"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683581    4494 cpu_manager.go:414] "containerMap: removing container" podUID="071face6-4616-498a-8f98-2ed795a18be5" containerName="storage-initializer"
Oct 30 03:34:55.683582 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683586    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071face6-4616-498a-8f98-2ed795a18be5" containerName="storage-initializer"
Oct 30 03:34:55.684066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683598    4494 cpu_manager.go:414] "containerMap: removing container" podUID="a83c2367-49ed-4fd9-878f-d13b3dd90bbb" containerName="storage-initializer"
Oct 30 03:34:55.684066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683603    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83c2367-49ed-4fd9-878f-d13b3dd90bbb" containerName="storage-initializer"
Oct 30 03:34:55.684066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683660    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy"
Oct 30 03:34:55.684066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683666    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a3121c-9c1e-4085-a4b6-151ccd0438e5" containerName="istio-proxy"
Oct 30 03:34:55.684066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683674    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e959be-b639-46ca-92d1-414217bceb87" containerName="main"
Oct 30 03:34:55.684066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683680    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83c2367-49ed-4fd9-878f-d13b3dd90bbb" containerName="main"
Oct 30 03:34:55.684066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.683684    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="071face6-4616-498a-8f98-2ed795a18be5" containerName="main"
Oct 30 03:34:55.688426 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.688406    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.705378 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.705328    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Oct 30 03:34:55.705491 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.705360    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:34:55.705491 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.705435    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Oct 30 03:34:55.705491 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.705487    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-tplmt\""
Oct 30 03:34:55.717433 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.717410    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Oct 30 03:34:55.717596 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.717444    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Oct 30 03:34:55.720176 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.720155    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"]
Oct 30 03:34:55.787287 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.787263    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.787432 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.787332    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-dshm\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.787432 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.787377    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.787534 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.787476    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-model-cache\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.787591 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.787526    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-home\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.787591 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.787550    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.787701 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.787626    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.787701 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.787672    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8bcg\" (UniqueName: \"kubernetes.io/projected/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kube-api-access-z8bcg\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.888327 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888300    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.888439 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888337    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-dshm\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.888439 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888353    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.888560 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888480    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-model-cache\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.888560 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888537    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-home\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.888668 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888575    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.888668 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888621    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.888778 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888671    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8bcg\" (UniqueName: \"kubernetes.io/projected/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kube-api-access-z8bcg\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.888870 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888847    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-model-cache\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.888944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888932    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.889004 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.888979    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-home\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.889219 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.889193    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.889407 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.889385    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.890785 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.890767    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-dshm\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.890931 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.890913    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:55.924733 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:55.924700    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8bcg\" (UniqueName: \"kubernetes.io/projected/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kube-api-access-z8bcg\") pod \"tokenizer-migration-path-test-kserve-bd846f858-q42ws\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:56.004420 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.004348    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:34:56.079720 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.079671    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"]
Oct 30 03:34:56.084406 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.084386    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.105923 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.105877    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"]
Oct 30 03:34:56.190972 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.190940    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.191152 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.190984    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.191152 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.191018    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.191152 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.191097    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.292435 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.292406    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.292581 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.292438    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.292581 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.292506    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.292581 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.292535    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.292870 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.292840    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.292946 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.292887    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.293155 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.293138    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.295083 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.295066    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.310668 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.310647    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"]
Oct 30 03:34:56.310837 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:34:56.310817    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ebeda1_ae0c_4487_b90b_80f08f7a01f6.slice/crio-a652bfa8924d6f3c800d793a8536d10b4463f91fe2ef7bc9219986f3ef684a9b WatchSource:0}: Error finding container a652bfa8924d6f3c800d793a8536d10b4463f91fe2ef7bc9219986f3ef684a9b: Status 404 returned error can't find the container with id a652bfa8924d6f3c800d793a8536d10b4463f91fe2ef7bc9219986f3ef684a9b
Oct 30 03:34:56.397046 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.397018    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" event={"ID":"81ebeda1-ae0c-4487-b90b-80f08f7a01f6","Type":"ContainerStarted","Data":"03270138804b12e9f78e91f2b43de385317cd9a0ea868decde6ad3329bdc2f0c"}
Oct 30 03:34:56.397162 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.397066    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" event={"ID":"81ebeda1-ae0c-4487-b90b-80f08f7a01f6","Type":"ContainerStarted","Data":"a652bfa8924d6f3c800d793a8536d10b4463f91fe2ef7bc9219986f3ef684a9b"}
Oct 30 03:34:56.403448 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.403426    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:34:56.700387 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:34:56.700336    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7652bd9_d3c3_48a3_bd3d_dfc27841d93e.slice/crio-6511c6b597f39910dad43df1876fc1eddc0866a394b3accf9f5e87549d7ba33c WatchSource:0}: Error finding container 6511c6b597f39910dad43df1876fc1eddc0866a394b3accf9f5e87549d7ba33c: Status 404 returned error can't find the container with id 6511c6b597f39910dad43df1876fc1eddc0866a394b3accf9f5e87549d7ba33c
Oct 30 03:34:56.700729 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:56.700560    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"]
Oct 30 03:34:57.405141 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:57.405104    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w" event={"ID":"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e","Type":"ContainerStarted","Data":"57cd6f7005983653994ef99579400a4fb41fbae52c7fa49b9238117d0fb71903"}
Oct 30 03:34:57.405323 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:57.405194    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w" event={"ID":"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e","Type":"ContainerStarted","Data":"6511c6b597f39910dad43df1876fc1eddc0866a394b3accf9f5e87549d7ba33c"}
Oct 30 03:34:59.425693 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:59.425654    4494 generic.go:356] "Generic (PLEG): container finished" podID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" containerID="03270138804b12e9f78e91f2b43de385317cd9a0ea868decde6ad3329bdc2f0c" exitCode=0
Oct 30 03:34:59.426181 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:59.425729    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" event={"ID":"81ebeda1-ae0c-4487-b90b-80f08f7a01f6","Type":"ContainerDied","Data":"03270138804b12e9f78e91f2b43de385317cd9a0ea868decde6ad3329bdc2f0c"}
Oct 30 03:34:59.427434 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:59.427407    4494 generic.go:356] "Generic (PLEG): container finished" podID="a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" containerID="57cd6f7005983653994ef99579400a4fb41fbae52c7fa49b9238117d0fb71903" exitCode=0
Oct 30 03:34:59.427525 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:34:59.427438    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w" event={"ID":"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e","Type":"ContainerDied","Data":"57cd6f7005983653994ef99579400a4fb41fbae52c7fa49b9238117d0fb71903"}
Oct 30 03:35:00.434575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:00.434542    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w" event={"ID":"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e","Type":"ContainerStarted","Data":"7b146e5c7f20941c441990d93195e74da6d55b0e25a782b33b8605b13cb3186f"}
Oct 30 03:35:00.435104 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:00.434772    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:35:00.436648 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:00.436626    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" event={"ID":"81ebeda1-ae0c-4487-b90b-80f08f7a01f6","Type":"ContainerStarted","Data":"70aa98d33ec1d94e7cd5d37b4fccc7c3d446fcd7ab58c66f92cfb522fce24d3a"}
Oct 30 03:35:00.483928 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:00.480239    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w" podStartSLOduration=2.031088481 podStartE2EDuration="4.480218392s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.449129911s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:00.477918165 +0000 UTC m=+2364.684098386" watchObservedRunningTime="2026-10-30 03:35:00.480218392 +0000 UTC m=+2364.686398613"
Oct 30 03:35:00.524996 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:00.524943    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" podStartSLOduration=3.157041875 podStartE2EDuration="5.524927603s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.367885728s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:00.523405671 +0000 UTC m=+2364.729585887" watchObservedRunningTime="2026-10-30 03:35:00.524927603 +0000 UTC m=+2364.731107821"
Oct 30 03:35:01.005236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:01.005183    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:35:06.005295 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:06.005267    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:35:06.018946 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:06.018889    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:35:06.474956 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:06.474918    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:35:24.081276 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:35:24.081232    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:35:31.452664 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.452633    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:35:31.491230 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.491206    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"]
Oct 30 03:35:31.496239 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.496220    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.523333 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.523304    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"]
Oct 30 03:35:31.610700 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.610666    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.610700 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.610709    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.610978 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.610941    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.610978 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.610970    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f2bk\" (UniqueName: \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-kube-api-access-8f2bk\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.611100 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.610987    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.611100 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.611013    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4869afaa-7493-4149-a0b2-c3b0f4b69418-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.611228 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.611116    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.611228 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.611144    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.611228 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.611211    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712264 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712191    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712264 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712230    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2bk\" (UniqueName: \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-kube-api-access-8f2bk\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712264 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712255    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712498 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712280    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4869afaa-7493-4149-a0b2-c3b0f4b69418-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712498 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712325    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712498 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712350    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712498 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712373    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712498 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712439    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712498 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712474    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712799 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712605    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712799 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712696    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712942 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712797    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.712942 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.712804    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.713153 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.713134    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4869afaa-7493-4149-a0b2-c3b0f4b69418-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.714784 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.714762    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.715284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.715267    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.735853 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.735825    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.736064 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.736042    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2bk\" (UniqueName: \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-kube-api-access-8f2bk\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:31.811352 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:31.811329    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:32.113768 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:35:32.113739    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4869afaa_7493_4149_a0b2_c3b0f4b69418.slice/crio-e2ff873d839287fc6744613e02d79a38ee21fc58d562c955431b7bc754d31791 WatchSource:0}: Error finding container e2ff873d839287fc6744613e02d79a38ee21fc58d562c955431b7bc754d31791: Status 404 returned error can't find the container with id e2ff873d839287fc6744613e02d79a38ee21fc58d562c955431b7bc754d31791
Oct 30 03:35:32.114385 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:32.114295    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"]
Oct 30 03:35:32.115911 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:32.115870    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:35:32.115972 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:32.115950    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:35:32.118227 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:32.116438    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:35:32.605995 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:32.605957    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" event={"ID":"4869afaa-7493-4149-a0b2-c3b0f4b69418","Type":"ContainerStarted","Data":"971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c"}
Oct 30 03:35:32.606472 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:32.606036    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" event={"ID":"4869afaa-7493-4149-a0b2-c3b0f4b69418","Type":"ContainerStarted","Data":"e2ff873d839287fc6744613e02d79a38ee21fc58d562c955431b7bc754d31791"}
Oct 30 03:35:32.652972 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:32.652922    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" podStartSLOduration=1.6528908310000001 podStartE2EDuration="1.652890831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:32.65100827 +0000 UTC m=+2396.857188500" watchObservedRunningTime="2026-10-30 03:35:32.652890831 +0000 UTC m=+2396.859071050"
Oct 30 03:35:32.812167 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:32.812130    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:33.073772 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:33.073742    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w_a7652bd9-d3c3-48a3-bd3d-dfc27841d93e/main/0.log"
Oct 30 03:35:33.812961 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:33.812930    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:35:33.813404 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:33.812986    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" podUID="4869afaa-7493-4149-a0b2-c3b0f4b69418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 03:35:33.966369 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:33.966341    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"]
Oct 30 03:35:33.966684 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:33.966627    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w" podUID="a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" containerName="main" containerID="cri-o://7b146e5c7f20941c441990d93195e74da6d55b0e25a782b33b8605b13cb3186f" gracePeriod=30
Oct 30 03:35:33.991437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:33.979552    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"]
Oct 30 03:35:34.015267 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:35:34.015218    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="70aa98d33ec1d94e7cd5d37b4fccc7c3d446fcd7ab58c66f92cfb522fce24d3a" cmd=["/bin/sleep","15"]
Oct 30 03:35:34.015433 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.015274    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:35:34.015433 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:35:34.015288    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" podUID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" containerName="main" containerID="cri-o://70aa98d33ec1d94e7cd5d37b4fccc7c3d446fcd7ab58c66f92cfb522fce24d3a"
Oct 30 03:35:34.015433 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.015320    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" podUID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" containerName="main" containerID="cri-o://70aa98d33ec1d94e7cd5d37b4fccc7c3d446fcd7ab58c66f92cfb522fce24d3a" gracePeriod=60
Oct 30 03:35:34.464281 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.464248    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" podUID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Oct 30 03:35:34.618098 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.618061    4494 generic.go:356] "Generic (PLEG): container finished" podID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" containerID="70aa98d33ec1d94e7cd5d37b4fccc7c3d446fcd7ab58c66f92cfb522fce24d3a" exitCode=0
Oct 30 03:35:34.618366 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.618164    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" event={"ID":"81ebeda1-ae0c-4487-b90b-80f08f7a01f6","Type":"ContainerDied","Data":"70aa98d33ec1d94e7cd5d37b4fccc7c3d446fcd7ab58c66f92cfb522fce24d3a"}
Oct 30 03:35:34.620670 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.620641    4494 generic.go:356] "Generic (PLEG): container finished" podID="a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" containerID="7b146e5c7f20941c441990d93195e74da6d55b0e25a782b33b8605b13cb3186f" exitCode=0
Oct 30 03:35:34.620784 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.620708    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w" event={"ID":"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e","Type":"ContainerDied","Data":"7b146e5c7f20941c441990d93195e74da6d55b0e25a782b33b8605b13cb3186f"}
Oct 30 03:35:34.717397 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.715577    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:35:34.808679 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.808649    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:35:34.811781 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.811757    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:35:34.811944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.811808    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" podUID="4869afaa-7493-4149-a0b2-c3b0f4b69418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 03:35:34.838309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.838282    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tls-certs\") pod \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") "
Oct 30 03:35:34.838691 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.838331    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-cabundle-cert\") pod \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") "
Oct 30 03:35:34.838691 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.838376    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-kserve-provision-location\") pod \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") "
Oct 30 03:35:34.838691 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.838491    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tmp-dir\") pod \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\" (UID: \"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e\") "
Oct 30 03:35:34.838833 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.838779    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-cabundle-cert" pod "a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" (UID: "a7652bd9-d3c3-48a3-bd3d-dfc27841d93e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:35:34.838890 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.838872    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tmp-dir" pod "a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" (UID: "a7652bd9-d3c3-48a3-bd3d-dfc27841d93e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.840648 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.840628    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tls-certs" pod "a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" (UID: "a7652bd9-d3c3-48a3-bd3d-dfc27841d93e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:35:34.861035 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.861012    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-kserve-provision-location" pod "a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" (UID: "a7652bd9-d3c3-48a3-bd3d-dfc27841d93e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.939316 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939288    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-cabundle-cert\") pod \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") "
Oct 30 03:35:34.939469 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939324    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kserve-provision-location\") pod \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") "
Oct 30 03:35:34.939469 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939347    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kube-api-access-z8bcg\" (UniqueName: \"kubernetes.io/projected/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kube-api-access-z8bcg\") pod \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") "
Oct 30 03:35:34.939469 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939373    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-model-cache\") pod \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") "
Oct 30 03:35:34.939469 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939403    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tls-certs\") pod \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") "
Oct 30 03:35:34.939469 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939441    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-home\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-home\") pod \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") "
Oct 30 03:35:34.939469 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939457    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tmp-dir\") pod \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") "
Oct 30 03:35:34.939777 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939511    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-dshm\") pod \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\" (UID: \"81ebeda1-ae0c-4487-b90b-80f08f7a01f6\") "
Oct 30 03:35:34.939777 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939672    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.939777 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939689    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.939777 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939704    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.939777 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939708    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-model-cache" pod "81ebeda1-ae0c-4487-b90b-80f08f7a01f6" (UID: "81ebeda1-ae0c-4487-b90b-80f08f7a01f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.939777 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939716    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-cabundle-cert" pod "81ebeda1-ae0c-4487-b90b-80f08f7a01f6" (UID: "81ebeda1-ae0c-4487-b90b-80f08f7a01f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:35:34.939777 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939718    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:34.940309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.939930    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-home" pod "81ebeda1-ae0c-4487-b90b-80f08f7a01f6" (UID: "81ebeda1-ae0c-4487-b90b-80f08f7a01f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.940309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.940058    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tmp-dir" pod "81ebeda1-ae0c-4487-b90b-80f08f7a01f6" (UID: "81ebeda1-ae0c-4487-b90b-80f08f7a01f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.941732 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.941704    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tls-certs" pod "81ebeda1-ae0c-4487-b90b-80f08f7a01f6" (UID: "81ebeda1-ae0c-4487-b90b-80f08f7a01f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:35:34.941825 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.941728    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kube-api-access-z8bcg" pod "81ebeda1-ae0c-4487-b90b-80f08f7a01f6" (UID: "81ebeda1-ae0c-4487-b90b-80f08f7a01f6"). InnerVolumeSpecName "kube-api-access-z8bcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:35:34.941919 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.941887    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-dshm" pod "81ebeda1-ae0c-4487-b90b-80f08f7a01f6" (UID: "81ebeda1-ae0c-4487-b90b-80f08f7a01f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:34.961408 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:34.961385    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kserve-provision-location" pod "81ebeda1-ae0c-4487-b90b-80f08f7a01f6" (UID: "81ebeda1-ae0c-4487-b90b-80f08f7a01f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:35.040409 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.040383    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:35.040409 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.040404    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:35.040409 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.040413    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:35.040590 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.040423    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:35.040590 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.040434    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:35.040590 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.040442    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8bcg\" (UniqueName: \"kubernetes.io/projected/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-kube-api-access-z8bcg\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:35.040590 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.040450    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:35.040590 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.040459    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ebeda1-ae0c-4487-b90b-80f08f7a01f6-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:35:35.627751 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.627724    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"
Oct 30 03:35:35.627951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.627722    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws" event={"ID":"81ebeda1-ae0c-4487-b90b-80f08f7a01f6","Type":"ContainerDied","Data":"a652bfa8924d6f3c800d793a8536d10b4463f91fe2ef7bc9219986f3ef684a9b"}
Oct 30 03:35:35.627951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.627866    4494 scope.go:122] "RemoveContainer" containerID="70aa98d33ec1d94e7cd5d37b4fccc7c3d446fcd7ab58c66f92cfb522fce24d3a"
Oct 30 03:35:35.629565 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.629540    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w" event={"ID":"a7652bd9-d3c3-48a3-bd3d-dfc27841d93e","Type":"ContainerDied","Data":"6511c6b597f39910dad43df1876fc1eddc0866a394b3accf9f5e87549d7ba33c"}
Oct 30 03:35:35.629565 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.629562    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"
Oct 30 03:35:35.637042 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.637019    4494 scope.go:122] "RemoveContainer" containerID="03270138804b12e9f78e91f2b43de385317cd9a0ea868decde6ad3329bdc2f0c"
Oct 30 03:35:35.644917 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.644885    4494 scope.go:122] "RemoveContainer" containerID="7b146e5c7f20941c441990d93195e74da6d55b0e25a782b33b8605b13cb3186f"
Oct 30 03:35:35.652060 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.652046    4494 scope.go:122] "RemoveContainer" containerID="57cd6f7005983653994ef99579400a4fb41fbae52c7fa49b9238117d0fb71903"
Oct 30 03:35:35.703922 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.703580    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"]
Oct 30 03:35:35.707022 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.707002    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-bd846f858-q42ws"]
Oct 30 03:35:35.743667 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.743640    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"]
Oct 30 03:35:35.753598 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.748825    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-67d4bcc576-7zv6w"]
Oct 30 03:35:35.812379 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.812347    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:35:35.812513 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.812470    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" podUID="4869afaa-7493-4149-a0b2-c3b0f4b69418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 03:35:35.927276 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.927187    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" path="/var/lib/kubelet/pods/81ebeda1-ae0c-4487-b90b-80f08f7a01f6/volumes"
Oct 30 03:35:35.927802 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:35.927780    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" path="/var/lib/kubelet/pods/a7652bd9-d3c3-48a3-bd3d-dfc27841d93e/volumes"
Oct 30 03:35:36.812504 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:36.812468    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:35:36.812680 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:36.812519    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" podUID="4869afaa-7493-4149-a0b2-c3b0f4b69418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 03:35:37.812266 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:37.812225    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 03:35:37.812727 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:37.812285    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" podUID="4869afaa-7493-4149-a0b2-c3b0f4b69418" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 03:35:38.815810 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:38.815784    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:38.816271 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:38.816109    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:38.816702 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:38.816683    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:35:46.526357 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526322    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"]
Oct 30 03:35:46.526789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526676    4494 cpu_manager.go:414] "containerMap: removing container" podUID="a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" containerName="main"
Oct 30 03:35:46.526789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526687    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" containerName="main"
Oct 30 03:35:46.526789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526709    4494 cpu_manager.go:414] "containerMap: removing container" podUID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" containerName="storage-initializer"
Oct 30 03:35:46.526789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526714    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" containerName="storage-initializer"
Oct 30 03:35:46.526789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526722    4494 cpu_manager.go:414] "containerMap: removing container" podUID="a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" containerName="storage-initializer"
Oct 30 03:35:46.526789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526726    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" containerName="storage-initializer"
Oct 30 03:35:46.526789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526734    4494 cpu_manager.go:414] "containerMap: removing container" podUID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" containerName="main"
Oct 30 03:35:46.526789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526739    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" containerName="main"
Oct 30 03:35:46.526789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526792    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7652bd9-d3c3-48a3-bd3d-dfc27841d93e" containerName="main"
Oct 30 03:35:46.527110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.526803    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ebeda1-ae0c-4487-b90b-80f08f7a01f6" containerName="main"
Oct 30 03:35:46.529933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.529891    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.542388 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.542369    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"]
Oct 30 03:35:46.545678 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.545663    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.558263 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.558245    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"]
Oct 30 03:35:46.571939 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.571916    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"]
Oct 30 03:35:46.637892 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.637867    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hws\" (UniqueName: \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-kube-api-access-48hws\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.638051 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.637914    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.638051 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.637943    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.638051 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638020    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.638051 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638050    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.638208 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638069    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.638208 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638178    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.638208 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638203    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvx57\" (UniqueName: \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-kube-api-access-rvx57\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.638313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638221    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.638313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638287    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.638313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638303    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.638430 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638319    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.638430 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638333    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.638430 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638391    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.638430 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638422    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.638578 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638463    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.638578 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638507    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.638578 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.638533    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.739103 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739077    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.739103 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739106    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.739284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739129    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.739284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739165    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.739284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739209    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hws\" (UniqueName: \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-kube-api-access-48hws\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.739284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739238    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.739284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739259    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.739557 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739296    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.739557 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739324    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.739557 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739354    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.739557 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739403    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.739557 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739431    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvx57\" (UniqueName: \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-kube-api-access-rvx57\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.739557 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739453    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.739557 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739500    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.739557 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739523    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.740005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739572    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.740005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739576    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.740005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739612    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.740005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739787    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.740005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739792    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.740005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739850    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.740005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739879    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.740005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739885    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.740005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.739963    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.740471 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.740191    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.740471 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.740361    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.740684 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.740633    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.740753 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.740709    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.742141 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.742120    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.742233 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.742204    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.742356 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.742335    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.742604 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.742587    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.764660 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.764629    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.764948 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.764929    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvx57\" (UniqueName: \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-kube-api-access-rvx57\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.765083 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.765058    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:46.765146 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.765118    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hws\" (UniqueName: \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-kube-api-access-48hws\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.843125 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.843059    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:46.857365 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:46.857340    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:47.161256 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:35:47.161232    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd73cd4_c05f_441d_b9ba_556eeb50c1aa.slice/crio-1c1474384f271eadda2a46f9b9886c8a0960a1a571d04244e55d949dceb77b37 WatchSource:0}: Error finding container 1c1474384f271eadda2a46f9b9886c8a0960a1a571d04244e55d949dceb77b37: Status 404 returned error can't find the container with id 1c1474384f271eadda2a46f9b9886c8a0960a1a571d04244e55d949dceb77b37
Oct 30 03:35:47.162518 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.162486    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"]
Oct 30 03:35:47.163695 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.163659    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:35:47.163794 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.163738    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:35:47.163794 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.163783    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:35:47.164037 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:35:47.164010    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47395de4_fb30_4ad7_bf0e_14fcd9a471bb.slice/crio-583b6e902eb2a3e0bb9745f179f4f491d88b1a2598b7636b699dad6657335856 WatchSource:0}: Error finding container 583b6e902eb2a3e0bb9745f179f4f491d88b1a2598b7636b699dad6657335856: Status 404 returned error can't find the container with id 583b6e902eb2a3e0bb9745f179f4f491d88b1a2598b7636b699dad6657335856
Oct 30 03:35:47.164667 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.164649    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"]
Oct 30 03:35:47.166085 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.166060    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:35:47.166135 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.166112    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:35:47.166181 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.166139    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:35:47.679711 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.679676    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" event={"ID":"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa","Type":"ContainerStarted","Data":"cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2"}
Oct 30 03:35:47.680228 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.679735    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" event={"ID":"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa","Type":"ContainerStarted","Data":"1c1474384f271eadda2a46f9b9886c8a0960a1a571d04244e55d949dceb77b37"}
Oct 30 03:35:47.681364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.681339    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" event={"ID":"47395de4-fb30-4ad7-bf0e-14fcd9a471bb","Type":"ContainerStarted","Data":"3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef"}
Oct 30 03:35:47.681496 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.681386    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" event={"ID":"47395de4-fb30-4ad7-bf0e-14fcd9a471bb","Type":"ContainerStarted","Data":"583b6e902eb2a3e0bb9745f179f4f491d88b1a2598b7636b699dad6657335856"}
Oct 30 03:35:47.725239 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.725203    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" podStartSLOduration=1.7251910430000001 podStartE2EDuration="1.725191043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:47.724347163 +0000 UTC m=+2411.930527384" watchObservedRunningTime="2026-10-30 03:35:47.725191043 +0000 UTC m=+2411.931371262"
Oct 30 03:35:47.770106 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.770065    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" podStartSLOduration=1.770052045 podStartE2EDuration="1.770052045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:47.768550047 +0000 UTC m=+2411.974730290" watchObservedRunningTime="2026-10-30 03:35:47.770052045 +0000 UTC m=+2411.976232262"
Oct 30 03:35:47.843552 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.843530    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:47.858232 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:47.858214    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:48.848226 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:48.848194    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:48.862806 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:48.862779    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:49.689702 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:49.689670    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:49.689952 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:49.689727    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:49.690580 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:49.690561    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:35:49.690693 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:35:49.690678    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:35:54.081993 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:35:54.081953    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:36:24.081838 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:36:24.081790    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:36:54.082178 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:36:54.082086    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:37:24.083113 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:37:24.083068    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:37:44.551978 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.551943    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"]
Oct 30 03:37:44.558563 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.558528    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.575617 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.575585    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:37:44.575757 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.575629    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Oct 30 03:37:44.575757 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.575695    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Oct 30 03:37:44.587444 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.587417    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Oct 30 03:37:44.587581 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.587447    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Oct 30 03:37:44.587581 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.587468    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-hm8vv\""
Oct 30 03:37:44.590047 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.590024    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"]
Oct 30 03:37:44.595451 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.595425    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.595577 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.595468    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-home\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.595577 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.595496    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dd46f7b-49f9-411a-9313-f228133f89d7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.595577 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.595553    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5762\" (UniqueName: \"kubernetes.io/projected/6dd46f7b-49f9-411a-9313-f228133f89d7-kube-api-access-r5762\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.595712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.595637    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-dshm\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.595712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.595662    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.595712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.595677    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-model-cache\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.595869 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.595771    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dd46f7b-49f9-411a-9313-f228133f89d7-tls-certs\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.696608 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.696564    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.696809 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.696619    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-home\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.696809 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.696647    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dd46f7b-49f9-411a-9313-f228133f89d7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.696809 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.696678    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5762\" (UniqueName: \"kubernetes.io/projected/6dd46f7b-49f9-411a-9313-f228133f89d7-kube-api-access-r5762\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.696809 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.696702    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-dshm\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.696809 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.696728    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.696809 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.696751    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-model-cache\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.696809 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.696803    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dd46f7b-49f9-411a-9313-f228133f89d7-tls-certs\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.697210 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.697115    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.697534 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.697504    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dd46f7b-49f9-411a-9313-f228133f89d7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.697823 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.697769    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-model-cache\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.697985 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.697964    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.698397 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.698370    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-home\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.699757 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.699726    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dd46f7b-49f9-411a-9313-f228133f89d7-tls-certs\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.699873 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.699848    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-dshm\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.721626 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.721593    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5762\" (UniqueName: \"kubernetes.io/projected/6dd46f7b-49f9-411a-9313-f228133f89d7-kube-api-access-r5762\") pod \"tokenizer-template-args-test-kserve-76c447b64-2v2jq\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.876369 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.876269    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:44.879544 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.879504    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"]
Oct 30 03:37:44.883193 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.883173    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:44.898847 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.898650    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:44.898847 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.898708    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:44.898847 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.898815    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05044ebc-4224-4ce2-9521-52c875007cc8-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:44.899142 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.898890    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05044ebc-4224-4ce2-9521-52c875007cc8-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:44.907176 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.907145    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"]
Oct 30 03:37:44.999611 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.999576    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:44.999806 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.999626    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:44.999806 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.999644    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05044ebc-4224-4ce2-9521-52c875007cc8-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:44.999806 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:44.999662    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05044ebc-4224-4ce2-9521-52c875007cc8-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:45.000092 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:45.000070    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:45.000132 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:45.000088    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:45.000401 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:45.000385    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05044ebc-4224-4ce2-9521-52c875007cc8-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:45.002260 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:45.002241    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05044ebc-4224-4ce2-9521-52c875007cc8-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65974b5957-gcctw\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:45.191815 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:45.191785    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"]
Oct 30 03:37:45.192096 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:37:45.192072    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd46f7b_49f9_411a_9313_f228133f89d7.slice/crio-14de04949d2f0ef8a963293d7d96f89b9855877bfd7150000356133346f3535c WatchSource:0}: Error finding container 14de04949d2f0ef8a963293d7d96f89b9855877bfd7150000356133346f3535c: Status 404 returned error can't find the container with id 14de04949d2f0ef8a963293d7d96f89b9855877bfd7150000356133346f3535c
Oct 30 03:37:45.203232 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:45.203205    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" event={"ID":"6dd46f7b-49f9-411a-9313-f228133f89d7","Type":"ContainerStarted","Data":"14de04949d2f0ef8a963293d7d96f89b9855877bfd7150000356133346f3535c"}
Oct 30 03:37:45.217386 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:45.217358    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:45.517433 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:45.517403    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"]
Oct 30 03:37:45.518721 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:37:45.518686    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05044ebc_4224_4ce2_9521_52c875007cc8.slice/crio-5ed4ffca08bb061fccab11f7706393f13f861eb52295ddc8bf0e5572d746f662 WatchSource:0}: Error finding container 5ed4ffca08bb061fccab11f7706393f13f861eb52295ddc8bf0e5572d746f662: Status 404 returned error can't find the container with id 5ed4ffca08bb061fccab11f7706393f13f861eb52295ddc8bf0e5572d746f662
Oct 30 03:37:46.210716 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:46.210677    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw" event={"ID":"05044ebc-4224-4ce2-9521-52c875007cc8","Type":"ContainerStarted","Data":"4f2e723582db0c6df20f80a20d49aa5216dfe7e20fa47f60d8f2ddc9b5b90169"}
Oct 30 03:37:46.211287 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:46.210738    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw" event={"ID":"05044ebc-4224-4ce2-9521-52c875007cc8","Type":"ContainerStarted","Data":"5ed4ffca08bb061fccab11f7706393f13f861eb52295ddc8bf0e5572d746f662"}
Oct 30 03:37:46.212436 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:46.212410    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" event={"ID":"6dd46f7b-49f9-411a-9313-f228133f89d7","Type":"ContainerStarted","Data":"1a600f42cc010437e6cca578d838aa3087e51d5cf2ef9024bc1c6ba99e60522e"}
Oct 30 03:37:49.261937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:49.256740    4494 generic.go:356] "Generic (PLEG): container finished" podID="6dd46f7b-49f9-411a-9313-f228133f89d7" containerID="1a600f42cc010437e6cca578d838aa3087e51d5cf2ef9024bc1c6ba99e60522e" exitCode=0
Oct 30 03:37:49.261937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:49.256924    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" event={"ID":"6dd46f7b-49f9-411a-9313-f228133f89d7","Type":"ContainerDied","Data":"1a600f42cc010437e6cca578d838aa3087e51d5cf2ef9024bc1c6ba99e60522e"}
Oct 30 03:37:49.269952 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:49.269126    4494 generic.go:356] "Generic (PLEG): container finished" podID="05044ebc-4224-4ce2-9521-52c875007cc8" containerID="4f2e723582db0c6df20f80a20d49aa5216dfe7e20fa47f60d8f2ddc9b5b90169" exitCode=0
Oct 30 03:37:49.269952 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:49.269206    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw" event={"ID":"05044ebc-4224-4ce2-9521-52c875007cc8","Type":"ContainerDied","Data":"4f2e723582db0c6df20f80a20d49aa5216dfe7e20fa47f60d8f2ddc9b5b90169"}
Oct 30 03:37:50.287925 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:50.284976    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" event={"ID":"6dd46f7b-49f9-411a-9313-f228133f89d7","Type":"ContainerStarted","Data":"b53fc4eaaa40c0f5b1efb58eb55b28fe41b308cdd51171d9fa79ae3a3d7b97aa"}
Oct 30 03:37:50.294923 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:50.292621    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw" event={"ID":"05044ebc-4224-4ce2-9521-52c875007cc8","Type":"ContainerStarted","Data":"2c78a1b9dba4c94c8a32f6cd921e7eb924e8ba3f236a44d7b6886f9ab52e603b"}
Oct 30 03:37:50.294923 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:50.293312    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:37:50.347923 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:50.346842    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" podStartSLOduration=3.1525837 podStartE2EDuration="6.346821228s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.194237528s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:50.333096112 +0000 UTC m=+2534.539276360" watchObservedRunningTime="2026-10-30 03:37:50.346821228 +0000 UTC m=+2534.553001448"
Oct 30 03:37:50.877926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:50.877110    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:54.085819 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:37:54.085781    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:37:54.877331 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:54.877235    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:54.891546 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:54.891510    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:54.957230 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:54.955880    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw" podStartSLOduration=7.45547949 podStartE2EDuration="10.955854725s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.500375235s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:50.385200027 +0000 UTC m=+2534.591380442" watchObservedRunningTime="2026-10-30 03:37:54.955854725 +0000 UTC m=+2539.162034946"
Oct 30 03:37:55.376466 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:55.376431    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:37:57.468119 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:57.375097    4494 generic.go:356] "Generic (PLEG): container finished" podID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerID="cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2" exitCode=0
Oct 30 03:37:57.468119 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:57.375179    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" event={"ID":"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa","Type":"ContainerDied","Data":"cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2"}
Oct 30 03:37:57.468119 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:57.375513    4494 scope.go:122] "RemoveContainer" containerID="cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2"
Oct 30 03:37:57.468119 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:57.376563    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:37:57.468119 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:57.376627    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:37:57.468119 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:57.376657    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:37:58.616729 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:58.384978    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" event={"ID":"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa","Type":"ContainerStarted","Data":"0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129"}
Oct 30 03:37:58.616729 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:58.387767    4494 generic.go:356] "Generic (PLEG): container finished" podID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerID="3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef" exitCode=0
Oct 30 03:37:58.616729 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:58.387918    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" event={"ID":"47395de4-fb30-4ad7-bf0e-14fcd9a471bb","Type":"ContainerDied","Data":"3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef"}
Oct 30 03:37:58.616729 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:58.388378    4494 scope.go:122] "RemoveContainer" containerID="3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef"
Oct 30 03:37:58.616729 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:58.389788    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:37:58.616729 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:58.389880    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:37:58.616729 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:58.389949    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:37:58.843560 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:58.843518    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:37:58.843864 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:58.843836    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 03:37:58.843947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:58.843922    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 03:37:59.440134 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:59.440089    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" event={"ID":"47395de4-fb30-4ad7-bf0e-14fcd9a471bb","Type":"ContainerStarted","Data":"ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3"}
Oct 30 03:37:59.844157 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:59.844120    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 03:37:59.844570 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:59.844170    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 03:37:59.858355 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:59.858324    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:37:59.858560 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:59.858541    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 03:37:59.858604 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:37:59.858584    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 03:38:00.843757 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:00.843715    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 03:38:00.843968 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:00.843788    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 03:38:00.858269 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:00.858237    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 03:38:00.858584 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:00.858307    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 03:38:01.844488 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:01.844449    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 03:38:01.844674 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:01.844503    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 03:38:01.858439 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:01.858408    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 03:38:01.858777 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:01.858470    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 03:38:02.843648 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:02.843613    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 03:38:02.843851 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:02.843674    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 03:38:02.858585 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:02.858542    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 03:38:02.859008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:02.858601    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 03:38:03.847511 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:03.847478    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:38:03.847890 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:03.847868    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:38:03.848504 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:03.848480    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:38:03.857967 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:03.857943    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 03:38:03.858031 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:03.857991    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 03:38:04.690500 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:04.690459    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:38:04.862406 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:04.862379    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:38:05.472336 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:05.472306    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:38:12.953682 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:12.953639    4494 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal" podUID="868aae1459799721b266e5e297e80105" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:38:21.315292 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:21.315199    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw" podUID="05044ebc-4224-4ce2-9521-52c875007cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 03:38:24.086986 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:38:24.086932    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:38:31.323746 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:31.323712    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:38:33.469014 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:33.468976    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-65974b5957-gcctw_05044ebc-4224-4ce2-9521-52c875007cc8/main/0.log"
Oct 30 03:38:34.081576 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.081531    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"]
Oct 30 03:38:34.091433 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:38:34.091388    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="b53fc4eaaa40c0f5b1efb58eb55b28fe41b308cdd51171d9fa79ae3a3d7b97aa" cmd=["/bin/sleep","15"]
Oct 30 03:38:34.091583 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.091453    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:38:34.091583 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:38:34.091466    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" podUID="6dd46f7b-49f9-411a-9313-f228133f89d7" containerName="main" containerID="cri-o://b53fc4eaaa40c0f5b1efb58eb55b28fe41b308cdd51171d9fa79ae3a3d7b97aa"
Oct 30 03:38:34.091583 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.091521    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" podUID="6dd46f7b-49f9-411a-9313-f228133f89d7" containerName="main" containerID="cri-o://b53fc4eaaa40c0f5b1efb58eb55b28fe41b308cdd51171d9fa79ae3a3d7b97aa" gracePeriod=60
Oct 30 03:38:34.099603 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.099572    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"]
Oct 30 03:38:34.099805 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.099784    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw" podUID="05044ebc-4224-4ce2-9521-52c875007cc8" containerName="main" containerID="cri-o://2c78a1b9dba4c94c8a32f6cd921e7eb924e8ba3f236a44d7b6886f9ab52e603b" gracePeriod=30
Oct 30 03:38:34.363936 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.363417    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" podUID="6dd46f7b-49f9-411a-9313-f228133f89d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Oct 30 03:38:34.620477 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.620382    4494 generic.go:356] "Generic (PLEG): container finished" podID="05044ebc-4224-4ce2-9521-52c875007cc8" containerID="2c78a1b9dba4c94c8a32f6cd921e7eb924e8ba3f236a44d7b6886f9ab52e603b" exitCode=0
Oct 30 03:38:34.620477 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.620458    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw" event={"ID":"05044ebc-4224-4ce2-9521-52c875007cc8","Type":"ContainerDied","Data":"2c78a1b9dba4c94c8a32f6cd921e7eb924e8ba3f236a44d7b6886f9ab52e603b"}
Oct 30 03:38:34.622935 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.622882    4494 generic.go:356] "Generic (PLEG): container finished" podID="6dd46f7b-49f9-411a-9313-f228133f89d7" containerID="b53fc4eaaa40c0f5b1efb58eb55b28fe41b308cdd51171d9fa79ae3a3d7b97aa" exitCode=0
Oct 30 03:38:34.623067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.622950    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" event={"ID":"6dd46f7b-49f9-411a-9313-f228133f89d7","Type":"ContainerDied","Data":"b53fc4eaaa40c0f5b1efb58eb55b28fe41b308cdd51171d9fa79ae3a3d7b97aa"}
Oct 30 03:38:34.677506 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.677474    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:38:34.848313 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.848282    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:38:34.871661 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871596    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-tmp-dir\") pod \"6dd46f7b-49f9-411a-9313-f228133f89d7\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") "
Oct 30 03:38:34.871661 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871627    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05044ebc-4224-4ce2-9521-52c875007cc8-tls-certs\" (UniqueName: \"kubernetes.io/secret/05044ebc-4224-4ce2-9521-52c875007cc8-tls-certs\") pod \"05044ebc-4224-4ce2-9521-52c875007cc8\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") "
Oct 30 03:38:34.871661 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871651    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd46f7b-49f9-411a-9313-f228133f89d7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dd46f7b-49f9-411a-9313-f228133f89d7-cabundle-cert\") pod \"6dd46f7b-49f9-411a-9313-f228133f89d7\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") "
Oct 30 03:38:34.871947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871673    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-home\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-home\") pod \"6dd46f7b-49f9-411a-9313-f228133f89d7\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") "
Oct 30 03:38:34.871947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871721    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd46f7b-49f9-411a-9313-f228133f89d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/6dd46f7b-49f9-411a-9313-f228133f89d7-tls-certs\") pod \"6dd46f7b-49f9-411a-9313-f228133f89d7\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") "
Oct 30 03:38:34.871947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871737    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05044ebc-4224-4ce2-9521-52c875007cc8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05044ebc-4224-4ce2-9521-52c875007cc8-cabundle-cert\") pod \"05044ebc-4224-4ce2-9521-52c875007cc8\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") "
Oct 30 03:38:34.871947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871789    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-dshm\") pod \"6dd46f7b-49f9-411a-9313-f228133f89d7\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") "
Oct 30 03:38:34.871947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871821    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-model-cache\") pod \"6dd46f7b-49f9-411a-9313-f228133f89d7\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") "
Oct 30 03:38:34.871947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871868    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-kserve-provision-location\") pod \"05044ebc-4224-4ce2-9521-52c875007cc8\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") "
Oct 30 03:38:34.871947 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871925    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd46f7b-49f9-411a-9313-f228133f89d7-kube-api-access-r5762\" (UniqueName: \"kubernetes.io/projected/6dd46f7b-49f9-411a-9313-f228133f89d7-kube-api-access-r5762\") pod \"6dd46f7b-49f9-411a-9313-f228133f89d7\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") "
Oct 30 03:38:34.872311 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871956    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-tmp-dir" pod "6dd46f7b-49f9-411a-9313-f228133f89d7" (UID: "6dd46f7b-49f9-411a-9313-f228133f89d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.872311 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.871975    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-kserve-provision-location\") pod \"6dd46f7b-49f9-411a-9313-f228133f89d7\" (UID: \"6dd46f7b-49f9-411a-9313-f228133f89d7\") "
Oct 30 03:38:34.872311 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.872008    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-tmp-dir\") pod \"05044ebc-4224-4ce2-9521-52c875007cc8\" (UID: \"05044ebc-4224-4ce2-9521-52c875007cc8\") "
Oct 30 03:38:34.872311 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.872067    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd46f7b-49f9-411a-9313-f228133f89d7-cabundle-cert" pod "6dd46f7b-49f9-411a-9313-f228133f89d7" (UID: "6dd46f7b-49f9-411a-9313-f228133f89d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:38:34.872311 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.872158    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-model-cache" pod "6dd46f7b-49f9-411a-9313-f228133f89d7" (UID: "6dd46f7b-49f9-411a-9313-f228133f89d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.872311 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.872193    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05044ebc-4224-4ce2-9521-52c875007cc8-cabundle-cert" pod "05044ebc-4224-4ce2-9521-52c875007cc8" (UID: "05044ebc-4224-4ce2-9521-52c875007cc8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:38:34.872693 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.872405    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6dd46f7b-49f9-411a-9313-f228133f89d7-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.872693 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.872427    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05044ebc-4224-4ce2-9521-52c875007cc8-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.872693 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.872439    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.872693 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.872452    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.874256 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.874226    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-home" pod "6dd46f7b-49f9-411a-9313-f228133f89d7" (UID: "6dd46f7b-49f9-411a-9313-f228133f89d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.874510 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.874490    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-dshm" pod "6dd46f7b-49f9-411a-9313-f228133f89d7" (UID: "6dd46f7b-49f9-411a-9313-f228133f89d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.874602 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.874559    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05044ebc-4224-4ce2-9521-52c875007cc8-tls-certs" pod "05044ebc-4224-4ce2-9521-52c875007cc8" (UID: "05044ebc-4224-4ce2-9521-52c875007cc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:38:34.874750 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.874730    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd46f7b-49f9-411a-9313-f228133f89d7-tls-certs" pod "6dd46f7b-49f9-411a-9313-f228133f89d7" (UID: "6dd46f7b-49f9-411a-9313-f228133f89d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:38:34.874750 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.874732    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-tmp-dir" pod "05044ebc-4224-4ce2-9521-52c875007cc8" (UID: "05044ebc-4224-4ce2-9521-52c875007cc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.874846 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.874763    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd46f7b-49f9-411a-9313-f228133f89d7-kube-api-access-r5762" pod "6dd46f7b-49f9-411a-9313-f228133f89d7" (UID: "6dd46f7b-49f9-411a-9313-f228133f89d7"). InnerVolumeSpecName "kube-api-access-r5762". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:38:34.898710 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.898674    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-kserve-provision-location" pod "6dd46f7b-49f9-411a-9313-f228133f89d7" (UID: "6dd46f7b-49f9-411a-9313-f228133f89d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.900437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.900399    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-kserve-provision-location" pod "05044ebc-4224-4ce2-9521-52c875007cc8" (UID: "05044ebc-4224-4ce2-9521-52c875007cc8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:38:34.973726 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.973692    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dd46f7b-49f9-411a-9313-f228133f89d7-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.973726 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.973722    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.973726 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.973731    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.974013 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.973741    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5762\" (UniqueName: \"kubernetes.io/projected/6dd46f7b-49f9-411a-9313-f228133f89d7-kube-api-access-r5762\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.974013 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.973750    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.974013 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.973761    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05044ebc-4224-4ce2-9521-52c875007cc8-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.974013 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.973770    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05044ebc-4224-4ce2-9521-52c875007cc8-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:34.974013 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:34.973778    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dd46f7b-49f9-411a-9313-f228133f89d7-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:38:35.629149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.629114    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"
Oct 30 03:38:35.629149 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.629131    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq" event={"ID":"6dd46f7b-49f9-411a-9313-f228133f89d7","Type":"ContainerDied","Data":"14de04949d2f0ef8a963293d7d96f89b9855877bfd7150000356133346f3535c"}
Oct 30 03:38:35.629722 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.629201    4494 scope.go:122] "RemoveContainer" containerID="b53fc4eaaa40c0f5b1efb58eb55b28fe41b308cdd51171d9fa79ae3a3d7b97aa"
Oct 30 03:38:35.631021 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.630995    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw" event={"ID":"05044ebc-4224-4ce2-9521-52c875007cc8","Type":"ContainerDied","Data":"5ed4ffca08bb061fccab11f7706393f13f861eb52295ddc8bf0e5572d746f662"}
Oct 30 03:38:35.631162 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.631006    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"
Oct 30 03:38:35.640591 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.640573    4494 scope.go:122] "RemoveContainer" containerID="1a600f42cc010437e6cca578d838aa3087e51d5cf2ef9024bc1c6ba99e60522e"
Oct 30 03:38:35.649496 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.649479    4494 scope.go:122] "RemoveContainer" containerID="2c78a1b9dba4c94c8a32f6cd921e7eb924e8ba3f236a44d7b6886f9ab52e603b"
Oct 30 03:38:35.657611 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.657589    4494 scope.go:122] "RemoveContainer" containerID="4f2e723582db0c6df20f80a20d49aa5216dfe7e20fa47f60d8f2ddc9b5b90169"
Oct 30 03:38:35.680923 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.680881    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"]
Oct 30 03:38:35.693440 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.693408    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65974b5957-gcctw"]
Oct 30 03:38:35.719565 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.719531    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"]
Oct 30 03:38:35.725756 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.725729    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-76c447b64-2v2jq"]
Oct 30 03:38:35.927332 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.927255    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05044ebc-4224-4ce2-9521-52c875007cc8" path="/var/lib/kubelet/pods/05044ebc-4224-4ce2-9521-52c875007cc8/volumes"
Oct 30 03:38:35.927733 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:38:35.927719    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd46f7b-49f9-411a-9313-f228133f89d7" path="/var/lib/kubelet/pods/6dd46f7b-49f9-411a-9313-f228133f89d7/volumes"
Oct 30 03:38:54.082397 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:38:54.082356    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:39:24.087187 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:39:24.087148    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:39:30.864025 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.863995    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"]
Oct 30 03:39:30.865194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.864373    4494 cpu_manager.go:414] "containerMap: removing container" podUID="05044ebc-4224-4ce2-9521-52c875007cc8" containerName="main"
Oct 30 03:39:30.865194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.864385    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05044ebc-4224-4ce2-9521-52c875007cc8" containerName="main"
Oct 30 03:39:30.865194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.864395    4494 cpu_manager.go:414] "containerMap: removing container" podUID="6dd46f7b-49f9-411a-9313-f228133f89d7" containerName="storage-initializer"
Oct 30 03:39:30.865194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.864400    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd46f7b-49f9-411a-9313-f228133f89d7" containerName="storage-initializer"
Oct 30 03:39:30.865194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.864424    4494 cpu_manager.go:414] "containerMap: removing container" podUID="05044ebc-4224-4ce2-9521-52c875007cc8" containerName="storage-initializer"
Oct 30 03:39:30.865194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.864428    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05044ebc-4224-4ce2-9521-52c875007cc8" containerName="storage-initializer"
Oct 30 03:39:30.865194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.864441    4494 cpu_manager.go:414] "containerMap: removing container" podUID="6dd46f7b-49f9-411a-9313-f228133f89d7" containerName="main"
Oct 30 03:39:30.865194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.864446    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd46f7b-49f9-411a-9313-f228133f89d7" containerName="main"
Oct 30 03:39:30.865194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.864502    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd46f7b-49f9-411a-9313-f228133f89d7" containerName="main"
Oct 30 03:39:30.865194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.864510    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="05044ebc-4224-4ce2-9521-52c875007cc8" containerName="main"
Oct 30 03:39:30.867649 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.867627    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:30.896298 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.896259    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Oct 30 03:39:30.896298 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.896272    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Oct 30 03:39:30.896298 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.896271    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-6rtrh\""
Oct 30 03:39:30.896298 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.896267    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Oct 30 03:39:30.898066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.898042    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"]
Oct 30 03:39:30.988824 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.988786    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:30.989067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.988852    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:30.989067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.988966    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:30.989067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.989000    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ps5v\" (UniqueName: \"kubernetes.io/projected/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-kube-api-access-4ps5v\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:30.989067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.989034    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:30.989219 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:30.989102    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.090792 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.090751    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.091025 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.090814    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.091025 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.090837    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.091025 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.090855    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ps5v\" (UniqueName: \"kubernetes.io/projected/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-kube-api-access-4ps5v\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.091025 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.090930    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.091025 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.090961    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.091797 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.091741    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.091797 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.091793    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.092064 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.092040    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.093867 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.093834    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.094052 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.094029    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.118234 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.118156    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ps5v\" (UniqueName: \"kubernetes.io/projected/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-kube-api-access-4ps5v\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.182824 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.182786    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:31.479791 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:39:31.479753    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26c0e54_aa26_4616_b9d6_5cd13e60cdae.slice/crio-28f7f0fef7406063bd939abbdfa407a9b2a9f8a69a6b753a173420d5a6556ff7 WatchSource:0}: Error finding container 28f7f0fef7406063bd939abbdfa407a9b2a9f8a69a6b753a173420d5a6556ff7: Status 404 returned error can't find the container with id 28f7f0fef7406063bd939abbdfa407a9b2a9f8a69a6b753a173420d5a6556ff7
Oct 30 03:39:31.479979 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.479932    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"]
Oct 30 03:39:31.880487 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.880444    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh" event={"ID":"d26c0e54-aa26-4616-b9d6-5cd13e60cdae","Type":"ContainerStarted","Data":"460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d"}
Oct 30 03:39:31.880973 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.880532    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh" event={"ID":"d26c0e54-aa26-4616-b9d6-5cd13e60cdae","Type":"ContainerStarted","Data":"28f7f0fef7406063bd939abbdfa407a9b2a9f8a69a6b753a173420d5a6556ff7"}
Oct 30 03:39:31.922364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:31.922301    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh" podStartSLOduration=1.922285005 podStartE2EDuration="1.922285005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:31.922032154 +0000 UTC m=+2636.128212374" watchObservedRunningTime="2026-10-30 03:39:31.922285005 +0000 UTC m=+2636.128465221"
Oct 30 03:39:32.183871 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:32.183781    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:41.183787 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:41.183750    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:41.196348 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:41.196327    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:41.992712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:39:41.992676    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:39:54.082568 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:39:54.082482    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:40:06.064850 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.064813    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"]
Oct 30 03:40:06.074362 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:40:06.074320    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d" cmd=["/bin/sleep","15"]
Oct 30 03:40:06.074518 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.074379    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:40:06.074518 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:40:06.074396    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh" podUID="d26c0e54-aa26-4616-b9d6-5cd13e60cdae" containerName="main" containerID="cri-o://460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d"
Oct 30 03:40:06.074518 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.074449    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh" podUID="d26c0e54-aa26-4616-b9d6-5cd13e60cdae" containerName="main" containerID="cri-o://460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d" gracePeriod=60
Oct 30 03:40:06.833245 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.833215    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:40:06.978182 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978085    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-model-cache\") pod \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") "
Oct 30 03:40:06.978182 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978145    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tmp-dir\") pod \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") "
Oct 30 03:40:06.978389 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978248    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tls-certs\" (UniqueName: \"kubernetes.io/secret/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tls-certs\") pod \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") "
Oct 30 03:40:06.978389 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978278    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-home\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-home\") pod \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") "
Oct 30 03:40:06.978389 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978316    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-kube-api-access-4ps5v\" (UniqueName: \"kubernetes.io/projected/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-kube-api-access-4ps5v\") pod \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") "
Oct 30 03:40:06.978549 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978401    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tmp-dir" pod "d26c0e54-aa26-4616-b9d6-5cd13e60cdae" (UID: "d26c0e54-aa26-4616-b9d6-5cd13e60cdae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:06.978549 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978412    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-model-cache" pod "d26c0e54-aa26-4616-b9d6-5cd13e60cdae" (UID: "d26c0e54-aa26-4616-b9d6-5cd13e60cdae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:06.978549 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978470    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-dshm\") pod \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\" (UID: \"d26c0e54-aa26-4616-b9d6-5cd13e60cdae\") "
Oct 30 03:40:06.978549 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978541    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-home" pod "d26c0e54-aa26-4616-b9d6-5cd13e60cdae" (UID: "d26c0e54-aa26-4616-b9d6-5cd13e60cdae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:06.978799 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978779    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:40:06.978853 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978804    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:40:06.978853 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.978819    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:40:06.980763 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.980735    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-kube-api-access-4ps5v" pod "d26c0e54-aa26-4616-b9d6-5cd13e60cdae" (UID: "d26c0e54-aa26-4616-b9d6-5cd13e60cdae"). InnerVolumeSpecName "kube-api-access-4ps5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:06.980875 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.980766    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tls-certs" pod "d26c0e54-aa26-4616-b9d6-5cd13e60cdae" (UID: "d26c0e54-aa26-4616-b9d6-5cd13e60cdae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:06.980875 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:06.980773    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-dshm" pod "d26c0e54-aa26-4616-b9d6-5cd13e60cdae" (UID: "d26c0e54-aa26-4616-b9d6-5cd13e60cdae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:07.079415 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.079375    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ps5v\" (UniqueName: \"kubernetes.io/projected/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-kube-api-access-4ps5v\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:40:07.079415 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.079407    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:40:07.079415 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.079418    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26c0e54-aa26-4616-b9d6-5cd13e60cdae-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:40:07.091099 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.091063    4494 generic.go:356] "Generic (PLEG): container finished" podID="d26c0e54-aa26-4616-b9d6-5cd13e60cdae" containerID="460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d" exitCode=0
Oct 30 03:40:07.091281 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.091128    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"
Oct 30 03:40:07.091281 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.091160    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh" event={"ID":"d26c0e54-aa26-4616-b9d6-5cd13e60cdae","Type":"ContainerDied","Data":"460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d"}
Oct 30 03:40:07.091281 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.091225    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh" event={"ID":"d26c0e54-aa26-4616-b9d6-5cd13e60cdae","Type":"ContainerDied","Data":"28f7f0fef7406063bd939abbdfa407a9b2a9f8a69a6b753a173420d5a6556ff7"}
Oct 30 03:40:07.091281 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.091258    4494 scope.go:122] "RemoveContainer" containerID="460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d"
Oct 30 03:40:07.100874 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.100858    4494 scope.go:122] "RemoveContainer" containerID="460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d"
Oct 30 03:40:07.101124 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:40:07.101104    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d\": container with ID starting with 460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d not found: ID does not exist" containerID="460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d"
Oct 30 03:40:07.101175 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.101135    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d"} err="failed to get container status \"460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d\": rpc error: code = NotFound desc = could not find container \"460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d\": container with ID starting with 460f0f65325124a99fe2057e050a521d89aaa3e71745fe6f0929396a15e3ca6d not found: ID does not exist"
Oct 30 03:40:07.145298 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.145267    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"]
Oct 30 03:40:07.157560 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.157521    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-58fbd56d5-7n8rh"]
Oct 30 03:40:07.927218 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:40:07.927180    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26c0e54-aa26-4616-b9d6-5cd13e60cdae" path="/var/lib/kubelet/pods/d26c0e54-aa26-4616-b9d6-5cd13e60cdae/volumes"
Oct 30 03:40:24.081663 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:40:24.081622    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:40:54.081828 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:40:54.081789    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:41:01.255456 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.255412    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"]
Oct 30 03:41:01.256027 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.255805    4494 cpu_manager.go:414] "containerMap: removing container" podUID="d26c0e54-aa26-4616-b9d6-5cd13e60cdae" containerName="main"
Oct 30 03:41:01.256027 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.255817    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26c0e54-aa26-4616-b9d6-5cd13e60cdae" containerName="main"
Oct 30 03:41:01.256027 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.255890    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26c0e54-aa26-4616-b9d6-5cd13e60cdae" containerName="main"
Oct 30 03:41:01.260344 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.260318    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.274622 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.274591    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.274763 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.274630    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.274763 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.274647    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbwg8\" (UniqueName: \"kubernetes.io/projected/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-kube-api-access-rbwg8\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.274763 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.274685    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.274866 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.274803    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.274866 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.274830    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.291005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.290969    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Oct 30 03:41:01.291181 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.290975    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Oct 30 03:41:01.291181 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.291000    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-stn6p\""
Oct 30 03:41:01.291181 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.291098    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Oct 30 03:41:01.293729 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.293704    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"]
Oct 30 03:41:01.375619 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.375576    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.375830 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.375646    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.375830 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.375669    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.375830 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.375705    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.376012 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.375840    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.376012 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.375888    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwg8\" (UniqueName: \"kubernetes.io/projected/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-kube-api-access-rbwg8\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.376133 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.376111    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.376210 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.376189    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.376272 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.376236    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.378197 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.378173    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.378712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.378687    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.401351 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.401313    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwg8\" (UniqueName: \"kubernetes.io/projected/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-kube-api-access-rbwg8\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.573659 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.573617    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:01.869354 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:41:01.869321    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e09111_d7bf_46f9_ad8f_0d64d7d9e15d.slice/crio-1659da32ca2578cadfd3f679d3574e7ee9782e77e62c76228a34dcc81727a449 WatchSource:0}: Error finding container 1659da32ca2578cadfd3f679d3574e7ee9782e77e62c76228a34dcc81727a449: Status 404 returned error can't find the container with id 1659da32ca2578cadfd3f679d3574e7ee9782e77e62c76228a34dcc81727a449
Oct 30 03:41:01.869848 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:01.869822    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"]
Oct 30 03:41:02.344976 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:02.344938    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn" event={"ID":"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d","Type":"ContainerStarted","Data":"305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9"}
Oct 30 03:41:02.345483 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:02.344991    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn" event={"ID":"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d","Type":"ContainerStarted","Data":"1659da32ca2578cadfd3f679d3574e7ee9782e77e62c76228a34dcc81727a449"}
Oct 30 03:41:02.389980 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:02.389934    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn" podStartSLOduration=1.389920589 podStartE2EDuration="1.389920589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:02.388838193 +0000 UTC m=+2726.595018413" watchObservedRunningTime="2026-10-30 03:41:02.389920589 +0000 UTC m=+2726.596100807"
Oct 30 03:41:02.574276 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:02.574246    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:11.574495 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:11.574451    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:11.587413 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:11.587385    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:12.458704 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:12.458674    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:41:24.082091 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:41:24.081988    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:41:38.732124 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.732089    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"]
Oct 30 03:41:38.736234 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.736212    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.762749 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.762723    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Oct 30 03:41:38.765688 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.765643    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"]
Oct 30 03:41:38.833141 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.833094    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.833335 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.833159    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.833335 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.833240    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.833335 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.833282    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.833504 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.833432    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpg9c\" (UniqueName: \"kubernetes.io/projected/dfca83d5-34f2-43c7-96ab-4fa637e3535e-kube-api-access-hpg9c\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.833504 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.833469    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.934413 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.934374    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpg9c\" (UniqueName: \"kubernetes.io/projected/dfca83d5-34f2-43c7-96ab-4fa637e3535e-kube-api-access-hpg9c\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.934413 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.934416    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.934686 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.934553    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.934686 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.934626    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.934686 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.934663    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.934834 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.934704    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.934871 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.934825    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.934979 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.934951    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.935110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.935083    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.937093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.937070    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.937379 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.937359    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:38.962094 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:38.962066    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpg9c\" (UniqueName: \"kubernetes.io/projected/dfca83d5-34f2-43c7-96ab-4fa637e3535e-kube-api-access-hpg9c\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:39.049282 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:39.049242    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:39.345123 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:41:39.345090    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfca83d5_34f2_43c7_96ab_4fa637e3535e.slice/crio-60a826227d9ac3ecc325872e06ee38a08bfd388aea09e7c70f2395eb1da5a1a0 WatchSource:0}: Error finding container 60a826227d9ac3ecc325872e06ee38a08bfd388aea09e7c70f2395eb1da5a1a0: Status 404 returned error can't find the container with id 60a826227d9ac3ecc325872e06ee38a08bfd388aea09e7c70f2395eb1da5a1a0
Oct 30 03:41:39.345332 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:39.345315    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"]
Oct 30 03:41:39.583336 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:39.583299    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr" event={"ID":"dfca83d5-34f2-43c7-96ab-4fa637e3535e","Type":"ContainerStarted","Data":"9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1"}
Oct 30 03:41:39.583655 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:39.583381    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr" event={"ID":"dfca83d5-34f2-43c7-96ab-4fa637e3535e","Type":"ContainerStarted","Data":"60a826227d9ac3ecc325872e06ee38a08bfd388aea09e7c70f2395eb1da5a1a0"}
Oct 30 03:41:40.049405 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:40.049367    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:49.049923 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:49.049851    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:49.062666 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:49.062633    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:49.105982 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:49.105926    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr" podStartSLOduration=11.105890749 podStartE2EDuration="11.105890749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:39.624960485 +0000 UTC m=+2763.831140706" watchObservedRunningTime="2026-10-30 03:41:49.105890749 +0000 UTC m=+2773.312070967"
Oct 30 03:41:49.639603 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:41:49.639567    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:41:54.081809 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:41:54.081771    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:42:13.557962 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:13.557873    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"]
Oct 30 03:42:13.610920 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:42:13.610854    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1" cmd=["/bin/sleep","15"]
Oct 30 03:42:13.611246 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:13.611198    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:42:13.611404 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:42:13.611383    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr" podUID="dfca83d5-34f2-43c7-96ab-4fa637e3535e" containerName="main" containerID="cri-o://9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1"
Oct 30 03:42:13.611605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:13.611582    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr" podUID="dfca83d5-34f2-43c7-96ab-4fa637e3535e" containerName="main" containerID="cri-o://9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1" gracePeriod=60
Oct 30 03:42:13.664944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:13.655615    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr" podUID="dfca83d5-34f2-43c7-96ab-4fa637e3535e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 03:42:13.997122 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:13.997097    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:42:14.082140 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082109    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-home\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-home\") pod \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") "
Oct 30 03:42:14.082330 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082156    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfca83d5-34f2-43c7-96ab-4fa637e3535e-kube-api-access-hpg9c\" (UniqueName: \"kubernetes.io/projected/dfca83d5-34f2-43c7-96ab-4fa637e3535e-kube-api-access-hpg9c\") pod \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") "
Oct 30 03:42:14.082330 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082229    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-dshm\") pod \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") "
Oct 30 03:42:14.082330 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082282    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-model-cache\") pod \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") "
Oct 30 03:42:14.082529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082335    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tmp-dir\") pod \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") "
Oct 30 03:42:14.082529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082397    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tls-certs\") pod \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\" (UID: \"dfca83d5-34f2-43c7-96ab-4fa637e3535e\") "
Oct 30 03:42:14.082529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082417    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-home" pod "dfca83d5-34f2-43c7-96ab-4fa637e3535e" (UID: "dfca83d5-34f2-43c7-96ab-4fa637e3535e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:14.082671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082588    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-model-cache" pod "dfca83d5-34f2-43c7-96ab-4fa637e3535e" (UID: "dfca83d5-34f2-43c7-96ab-4fa637e3535e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:14.082671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082602    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tmp-dir" pod "dfca83d5-34f2-43c7-96ab-4fa637e3535e" (UID: "dfca83d5-34f2-43c7-96ab-4fa637e3535e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:14.082786 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082772    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:42:14.082840 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082788    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:42:14.082840 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.082797    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:42:14.084742 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.084715    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-dshm" pod "dfca83d5-34f2-43c7-96ab-4fa637e3535e" (UID: "dfca83d5-34f2-43c7-96ab-4fa637e3535e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:42:14.084742 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.084722    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfca83d5-34f2-43c7-96ab-4fa637e3535e-kube-api-access-hpg9c" pod "dfca83d5-34f2-43c7-96ab-4fa637e3535e" (UID: "dfca83d5-34f2-43c7-96ab-4fa637e3535e"). InnerVolumeSpecName "kube-api-access-hpg9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:42:14.084882 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.084817    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tls-certs" pod "dfca83d5-34f2-43c7-96ab-4fa637e3535e" (UID: "dfca83d5-34f2-43c7-96ab-4fa637e3535e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:42:14.183410 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.183318    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpg9c\" (UniqueName: \"kubernetes.io/projected/dfca83d5-34f2-43c7-96ab-4fa637e3535e-kube-api-access-hpg9c\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:42:14.183410 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.183346    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfca83d5-34f2-43c7-96ab-4fa637e3535e-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:42:14.183410 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.183355    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfca83d5-34f2-43c7-96ab-4fa637e3535e-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:42:14.763110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.763074    4494 generic.go:356] "Generic (PLEG): container finished" podID="dfca83d5-34f2-43c7-96ab-4fa637e3535e" containerID="9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1" exitCode=0
Oct 30 03:42:14.763627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.763129    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"
Oct 30 03:42:14.763627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.763155    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr" event={"ID":"dfca83d5-34f2-43c7-96ab-4fa637e3535e","Type":"ContainerDied","Data":"9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1"}
Oct 30 03:42:14.763627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.763209    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr" event={"ID":"dfca83d5-34f2-43c7-96ab-4fa637e3535e","Type":"ContainerDied","Data":"60a826227d9ac3ecc325872e06ee38a08bfd388aea09e7c70f2395eb1da5a1a0"}
Oct 30 03:42:14.763627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.763283    4494 scope.go:122] "RemoveContainer" containerID="9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1"
Oct 30 03:42:14.772839 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.772821    4494 scope.go:122] "RemoveContainer" containerID="9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1"
Oct 30 03:42:14.773145 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:42:14.773127    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1\": container with ID starting with 9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1 not found: ID does not exist" containerID="9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1"
Oct 30 03:42:14.773225 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.773155    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1"} err="failed to get container status \"9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1\": rpc error: code = NotFound desc = could not find container \"9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1\": container with ID starting with 9ea402d9f312991e84052c81873a4f8e1caa4b8b848868ff6cbf689427d5d3f1 not found: ID does not exist"
Oct 30 03:42:14.814355 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.814322    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"]
Oct 30 03:42:14.826416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:14.826390    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d7985584b5dtcr"]
Oct 30 03:42:15.926531 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:42:15.926498    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfca83d5-34f2-43c7-96ab-4fa637e3535e" path="/var/lib/kubelet/pods/dfca83d5-34f2-43c7-96ab-4fa637e3535e/volumes"
Oct 30 03:42:24.081436 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:42:24.081397    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:42:54.082722 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:42:54.082630    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:43:03.369360 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.369325    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"]
Oct 30 03:43:03.378700 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:43:03.378659    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9" cmd=["/bin/sleep","15"]
Oct 30 03:43:03.378849 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.378720    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:43:03.378849 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:43:03.378733    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn" podUID="d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" containerName="main" containerID="cri-o://305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9"
Oct 30 03:43:03.378849 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.378781    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn" podUID="d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" containerName="main" containerID="cri-o://305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9" gracePeriod=60
Oct 30 03:43:03.448411 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.448366    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn" podUID="d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 30 03:43:03.734421 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.734391    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:43:03.837909 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.837874    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-home\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-home\") pod \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") "
Oct 30 03:43:03.838089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.837949    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tmp-dir\") pod \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") "
Oct 30 03:43:03.838089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.837989    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-dshm\") pod \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") "
Oct 30 03:43:03.838089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.838020    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-model-cache\") pod \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") "
Oct 30 03:43:03.838270 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.838095    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tls-certs\") pod \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") "
Oct 30 03:43:03.838270 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.838142    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-home" pod "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" (UID: "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:03.838270 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.838171    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-kube-api-access-rbwg8\" (UniqueName: \"kubernetes.io/projected/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-kube-api-access-rbwg8\") pod \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\" (UID: \"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d\") "
Oct 30 03:43:03.838456 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.838294    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tmp-dir" pod "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" (UID: "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:03.838456 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.838346    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-model-cache" pod "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" (UID: "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:03.838541 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.838496    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:43:03.838541 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.838515    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:43:03.838541 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.838532    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:43:03.840459 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.840436    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-kube-api-access-rbwg8" pod "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" (UID: "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d"). InnerVolumeSpecName "kube-api-access-rbwg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:03.840802 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.840779    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tls-certs" pod "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" (UID: "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:03.840934 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.840781    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-dshm" pod "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" (UID: "d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:03.939669 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.939626    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:43:03.939669 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.939665    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbwg8\" (UniqueName: \"kubernetes.io/projected/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-kube-api-access-rbwg8\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:43:03.939669 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.939674    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:43:03.986518 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.986480    4494 generic.go:356] "Generic (PLEG): container finished" podID="d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" containerID="305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9" exitCode=0
Oct 30 03:43:03.986722 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.986535    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"
Oct 30 03:43:03.986722 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.986587    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn" event={"ID":"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d","Type":"ContainerDied","Data":"305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9"}
Oct 30 03:43:03.986722 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.986658    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn" event={"ID":"d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d","Type":"ContainerDied","Data":"1659da32ca2578cadfd3f679d3574e7ee9782e77e62c76228a34dcc81727a449"}
Oct 30 03:43:03.986722 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.986696    4494 scope.go:122] "RemoveContainer" containerID="305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9"
Oct 30 03:43:03.996296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.996272    4494 scope.go:122] "RemoveContainer" containerID="305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9"
Oct 30 03:43:03.996594 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:43:03.996572    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9\": container with ID starting with 305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9 not found: ID does not exist" containerID="305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9"
Oct 30 03:43:03.996650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:03.996605    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9"} err="failed to get container status \"305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9\": rpc error: code = NotFound desc = could not find container \"305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9\": container with ID starting with 305077f9d1e04ce3f073b7986c82b477a2b5c8b838a51f5975878af5708bb6a9 not found: ID does not exist"
Oct 30 03:43:04.036978 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:04.036946    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"]
Oct 30 03:43:04.049052 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:04.049026    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d5584858rmn"]
Oct 30 03:43:05.926726 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:43:05.926692    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" path="/var/lib/kubelet/pods/d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d/volumes"
Oct 30 03:43:24.082796 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:43:24.082757    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:43:54.081257 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:43:54.081214    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:44:02.667438 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.667402    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"]
Oct 30 03:44:02.667832 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.667776    4494 cpu_manager.go:414] "containerMap: removing container" podUID="dfca83d5-34f2-43c7-96ab-4fa637e3535e" containerName="main"
Oct 30 03:44:02.667832 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.667788    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfca83d5-34f2-43c7-96ab-4fa637e3535e" containerName="main"
Oct 30 03:44:02.667832 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.667807    4494 cpu_manager.go:414] "containerMap: removing container" podUID="d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" containerName="main"
Oct 30 03:44:02.667832 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.667812    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" containerName="main"
Oct 30 03:44:02.668005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.667875    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfca83d5-34f2-43c7-96ab-4fa637e3535e" containerName="main"
Oct 30 03:44:02.668005 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.667885    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e09111-d7bf-46f9-ad8f-0d64d7d9e15d" containerName="main"
Oct 30 03:44:02.672223 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.672203    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.702922 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.702880    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Oct 30 03:44:02.703102 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.703025    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-hpj55\""
Oct 30 03:44:02.703304 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.703289    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Oct 30 03:44:02.703386 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.703325    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Oct 30 03:44:02.706151 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.706130    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"]
Oct 30 03:44:02.772869 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.772827    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.773068 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.773012    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.773068 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.773041    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.773173 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.773071    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.773173 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.773099    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x76xp\" (UniqueName: \"kubernetes.io/projected/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-kube-api-access-x76xp\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.773248 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.773229    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.874468 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.874422    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.874681 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.874477    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.874681 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.874504    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.874681 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.874541    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.874681 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.874568    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x76xp\" (UniqueName: \"kubernetes.io/projected/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-kube-api-access-x76xp\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.874681 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.874607    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.875015 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.874991    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.875091 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.875021    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.875091 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.875048    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.877048 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.877026    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.877347 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.877330    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.898332 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.898307    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x76xp\" (UniqueName: \"kubernetes.io/projected/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-kube-api-access-x76xp\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:02.985254 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:02.985150    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:03.282867 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:03.282835    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"]
Oct 30 03:44:03.283255 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:44:03.283233    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e7cd1e_3a7c_46c3_8c31_5d1d361e43f8.slice/crio-e16ca458f73be4974a1fa303623999f3f6642075ac172d2ccc732634d7bf65b3 WatchSource:0}: Error finding container e16ca458f73be4974a1fa303623999f3f6642075ac172d2ccc732634d7bf65b3: Status 404 returned error can't find the container with id e16ca458f73be4974a1fa303623999f3f6642075ac172d2ccc732634d7bf65b3
Oct 30 03:44:04.255211 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:04.255171    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk" event={"ID":"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8","Type":"ContainerStarted","Data":"48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee"}
Oct 30 03:44:04.255671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:04.255229    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk" event={"ID":"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8","Type":"ContainerStarted","Data":"e16ca458f73be4974a1fa303623999f3f6642075ac172d2ccc732634d7bf65b3"}
Oct 30 03:44:04.298778 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:04.298719    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk" podStartSLOduration=2.298703595 podStartE2EDuration="2.298703595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:04.297492184 +0000 UTC m=+2908.503672401" watchObservedRunningTime="2026-10-30 03:44:04.298703595 +0000 UTC m=+2908.504883812"
Oct 30 03:44:04.985586 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:04.985534    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:12.986329 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:12.986290    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:12.999000 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:12.998963    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:13.314066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:13.314039    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:44:24.081953 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:44:24.081852    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:44:46.232792 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.232747    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"]
Oct 30 03:44:46.236590 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.236567    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.258611 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.258576    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"]
Oct 30 03:44:46.269060 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.269027    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Oct 30 03:44:46.291350 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.291305    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.291350 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.291349    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e6755d-a445-47c7-be8a-31e668a8f3fa-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.291590 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.291369    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7c22\" (UniqueName: \"kubernetes.io/projected/69e6755d-a445-47c7-be8a-31e668a8f3fa-kube-api-access-n7c22\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.291590 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.291454    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.291590 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.291560    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.291712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.291593    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.392848 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.392807    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.392848 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.392848    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e6755d-a445-47c7-be8a-31e668a8f3fa-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.393139 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.392870    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7c22\" (UniqueName: \"kubernetes.io/projected/69e6755d-a445-47c7-be8a-31e668a8f3fa-kube-api-access-n7c22\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.393139 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.392937    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.393139 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.392980    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.393139 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.393011    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.393340 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.393321    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.393469 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.393431    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.393592 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.393468    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.395592 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.395568    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.395864 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.395844    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e6755d-a445-47c7-be8a-31e668a8f3fa-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.416841 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.416805    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7c22\" (UniqueName: \"kubernetes.io/projected/69e6755d-a445-47c7-be8a-31e668a8f3fa-kube-api-access-n7c22\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.550700 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.550666    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:46.851160 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:44:46.851126    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e6755d_a445_47c7_be8a_31e668a8f3fa.slice/crio-b3ecfe12c412a3a4b9c699081fcb9ab33be5cb0d465b7166406fde8f7350e4f8 WatchSource:0}: Error finding container b3ecfe12c412a3a4b9c699081fcb9ab33be5cb0d465b7166406fde8f7350e4f8: Status 404 returned error can't find the container with id b3ecfe12c412a3a4b9c699081fcb9ab33be5cb0d465b7166406fde8f7350e4f8
Oct 30 03:44:46.851618 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:46.851594    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"]
Oct 30 03:44:47.460769 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:47.460731    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8" event={"ID":"69e6755d-a445-47c7-be8a-31e668a8f3fa","Type":"ContainerStarted","Data":"c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e"}
Oct 30 03:44:47.461186 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:47.460788    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8" event={"ID":"69e6755d-a445-47c7-be8a-31e668a8f3fa","Type":"ContainerStarted","Data":"b3ecfe12c412a3a4b9c699081fcb9ab33be5cb0d465b7166406fde8f7350e4f8"}
Oct 30 03:44:47.502951 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:47.502878    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8" podStartSLOduration=1.502863943 podStartE2EDuration="1.502863943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:47.501700316 +0000 UTC m=+2951.707880536" watchObservedRunningTime="2026-10-30 03:44:47.502863943 +0000 UTC m=+2951.709044162"
Oct 30 03:44:47.551820 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:47.551778    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:54.082387 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:44:54.082342    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:44:56.551777 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:56.551732    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:56.564336 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:56.564310    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:44:57.518600 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:44:57.518561    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:45:20.585657 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:20.585617    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"]
Oct 30 03:45:20.596417 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:45:20.596354    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e" cmd=["/bin/sleep","15"]
Oct 30 03:45:20.596660 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:20.596437    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:45:20.596660 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:45:20.596465    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8" podUID="69e6755d-a445-47c7-be8a-31e668a8f3fa" containerName="main" containerID="cri-o://c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e"
Oct 30 03:45:20.596660 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:20.596547    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8" podUID="69e6755d-a445-47c7-be8a-31e668a8f3fa" containerName="main" containerID="cri-o://c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e" gracePeriod=60
Oct 30 03:45:20.981720 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:20.981384    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:45:21.066760 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.066701    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-home\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-home\") pod \"69e6755d-a445-47c7-be8a-31e668a8f3fa\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") "
Oct 30 03:45:21.067000 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.066778    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-dshm\") pod \"69e6755d-a445-47c7-be8a-31e668a8f3fa\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") "
Oct 30 03:45:21.067000 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.066806    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-tmp-dir\") pod \"69e6755d-a445-47c7-be8a-31e668a8f3fa\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") "
Oct 30 03:45:21.067000 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.066921    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69e6755d-a445-47c7-be8a-31e668a8f3fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/69e6755d-a445-47c7-be8a-31e668a8f3fa-tls-certs\") pod \"69e6755d-a445-47c7-be8a-31e668a8f3fa\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") "
Oct 30 03:45:21.067000 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.066981    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-model-cache\") pod \"69e6755d-a445-47c7-be8a-31e668a8f3fa\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") "
Oct 30 03:45:21.067269 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.067022    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e6755d-a445-47c7-be8a-31e668a8f3fa-kube-api-access-n7c22\" (UniqueName: \"kubernetes.io/projected/69e6755d-a445-47c7-be8a-31e668a8f3fa-kube-api-access-n7c22\") pod \"69e6755d-a445-47c7-be8a-31e668a8f3fa\" (UID: \"69e6755d-a445-47c7-be8a-31e668a8f3fa\") "
Oct 30 03:45:21.067686 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.067641    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-model-cache" pod "69e6755d-a445-47c7-be8a-31e668a8f3fa" (UID: "69e6755d-a445-47c7-be8a-31e668a8f3fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:21.068157 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.068105    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-home" pod "69e6755d-a445-47c7-be8a-31e668a8f3fa" (UID: "69e6755d-a445-47c7-be8a-31e668a8f3fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:21.068743 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.068706    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-tmp-dir" pod "69e6755d-a445-47c7-be8a-31e668a8f3fa" (UID: "69e6755d-a445-47c7-be8a-31e668a8f3fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:21.071627 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.071593    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-dshm" pod "69e6755d-a445-47c7-be8a-31e668a8f3fa" (UID: "69e6755d-a445-47c7-be8a-31e668a8f3fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:21.074397 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.074365    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e6755d-a445-47c7-be8a-31e668a8f3fa-kube-api-access-n7c22" pod "69e6755d-a445-47c7-be8a-31e668a8f3fa" (UID: "69e6755d-a445-47c7-be8a-31e668a8f3fa"). InnerVolumeSpecName "kube-api-access-n7c22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:21.074509 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.074470    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e6755d-a445-47c7-be8a-31e668a8f3fa-tls-certs" pod "69e6755d-a445-47c7-be8a-31e668a8f3fa" (UID: "69e6755d-a445-47c7-be8a-31e668a8f3fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:21.168428 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.168388    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e6755d-a445-47c7-be8a-31e668a8f3fa-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.168428 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.168427    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.168639 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.168442    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7c22\" (UniqueName: \"kubernetes.io/projected/69e6755d-a445-47c7-be8a-31e668a8f3fa-kube-api-access-n7c22\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.168639 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.168455    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.168639 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.168466    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.168639 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.168476    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e6755d-a445-47c7-be8a-31e668a8f3fa-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:45:21.637031 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.636994    4494 generic.go:356] "Generic (PLEG): container finished" podID="69e6755d-a445-47c7-be8a-31e668a8f3fa" containerID="c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e" exitCode=0
Oct 30 03:45:21.637031 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.637033    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8" event={"ID":"69e6755d-a445-47c7-be8a-31e668a8f3fa","Type":"ContainerDied","Data":"c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e"}
Oct 30 03:45:21.637655 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.637055    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"
Oct 30 03:45:21.637655 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.637075    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8" event={"ID":"69e6755d-a445-47c7-be8a-31e668a8f3fa","Type":"ContainerDied","Data":"b3ecfe12c412a3a4b9c699081fcb9ab33be5cb0d465b7166406fde8f7350e4f8"}
Oct 30 03:45:21.637655 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.637097    4494 scope.go:122] "RemoveContainer" containerID="c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e"
Oct 30 03:45:21.646798 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.646775    4494 scope.go:122] "RemoveContainer" containerID="c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e"
Oct 30 03:45:21.647101 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:45:21.647068    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e\": container with ID starting with c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e not found: ID does not exist" containerID="c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e"
Oct 30 03:45:21.647184 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.647112    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e"} err="failed to get container status \"c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e\": rpc error: code = NotFound desc = could not find container \"c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e\": container with ID starting with c9736caf7791a2a6998b470e5dfce5005f311c878dfdfeac171b9a9936ef786e not found: ID does not exist"
Oct 30 03:45:21.688817 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.688781    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"]
Oct 30 03:45:21.701194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.701167    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f747777cnt8"]
Oct 30 03:45:21.927967 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:45:21.927865    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e6755d-a445-47c7-be8a-31e668a8f3fa" path="/var/lib/kubelet/pods/69e6755d-a445-47c7-be8a-31e668a8f3fa/volumes"
Oct 30 03:45:24.082656 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:45:24.082621    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:45:54.081134 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:45:54.081054    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:46:10.793677 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:10.793642    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"]
Oct 30 03:46:10.803108 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:46:10.803063    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee" cmd=["/bin/sleep","15"]
Oct 30 03:46:10.803246 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:10.803122    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 03:46:10.803246 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:46:10.803134    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk" podUID="28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" containerName="main" containerID="cri-o://48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee"
Oct 30 03:46:10.803246 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:10.803185    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk" podUID="28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" containerName="main" containerID="cri-o://48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee" gracePeriod=60
Oct 30 03:46:11.149084 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.149059    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:46:11.184920 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.184879    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tmp-dir\") pod \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") "
Oct 30 03:46:11.185064 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.184965    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-home\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-home\") pod \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") "
Oct 30 03:46:11.185064 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.185001    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-model-cache\") pod \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") "
Oct 30 03:46:11.185064 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.185045    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-dshm\") pod \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") "
Oct 30 03:46:11.185235 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.185081    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-kube-api-access-x76xp\" (UniqueName: \"kubernetes.io/projected/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-kube-api-access-x76xp\") pod \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") "
Oct 30 03:46:11.185235 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.185090    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tmp-dir" pod "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" (UID: "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:11.185235 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.185105    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tls-certs\") pod \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\" (UID: \"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8\") "
Oct 30 03:46:11.185365 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.185255    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:11.185365 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.185284    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-model-cache" pod "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" (UID: "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:11.185365 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.185297    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-home" pod "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" (UID: "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:11.187329 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.187302    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-kube-api-access-x76xp" pod "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" (UID: "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8"). InnerVolumeSpecName "kube-api-access-x76xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:11.187329 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.187324    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-dshm" pod "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" (UID: "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:11.187454 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.187335    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tls-certs" pod "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" (UID: "28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:11.286210 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.286181    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:11.286210 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.286205    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:11.286210 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.286216    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:11.286409 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.286226    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x76xp\" (UniqueName: \"kubernetes.io/projected/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-kube-api-access-x76xp\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:11.286409 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.286235    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:11.870995 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.870954    4494 generic.go:356] "Generic (PLEG): container finished" podID="28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" containerID="48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee" exitCode=0
Oct 30 03:46:11.871445 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.871001    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk" event={"ID":"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8","Type":"ContainerDied","Data":"48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee"}
Oct 30 03:46:11.871445 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.871017    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"
Oct 30 03:46:11.871445 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.871081    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk" event={"ID":"28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8","Type":"ContainerDied","Data":"e16ca458f73be4974a1fa303623999f3f6642075ac172d2ccc732634d7bf65b3"}
Oct 30 03:46:11.871445 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.871113    4494 scope.go:122] "RemoveContainer" containerID="48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee"
Oct 30 03:46:11.880806 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.880786    4494 scope.go:122] "RemoveContainer" containerID="48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee"
Oct 30 03:46:11.881090 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:46:11.881070    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee\": container with ID starting with 48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee not found: ID does not exist" containerID="48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee"
Oct 30 03:46:11.881132 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.881100    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee"} err="failed to get container status \"48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee\": rpc error: code = NotFound desc = could not find container \"48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee\": container with ID starting with 48de36cefdb9c2e25b5bdc11b1dcbfbff60e32dd5836db4955ac6b5060334aee not found: ID does not exist"
Oct 30 03:46:11.926701 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.926673    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"]
Oct 30 03:46:11.934163 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:11.934140    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-mwllk"]
Oct 30 03:46:12.954621 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:12.954584    4494 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-130.ec2.internal" podUID="868aae1459799721b266e5e297e80105" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:46:13.928063 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:13.928025    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" path="/var/lib/kubelet/pods/28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8/volumes"
Oct 30 03:46:24.081190 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:46:24.081152    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:46:48.442689 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:48.442648    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"]
Oct 30 03:46:48.443269 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:48.442865    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy" containerID="cri-o://0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129" gracePeriod=30
Oct 30 03:46:48.444478 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:48.444452    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"]
Oct 30 03:46:48.444658 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:48.444624    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy" containerID="cri-o://ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3" gracePeriod=30
Oct 30 03:46:48.445791 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:48.445767    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"]
Oct 30 03:46:48.446015 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:48.445974    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" podUID="4869afaa-7493-4149-a0b2-c3b0f4b69418" containerName="istio-proxy" containerID="cri-o://971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c" gracePeriod=30
Oct 30 03:46:48.850109 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:48.850077    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:46:48.850269 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:48.850127    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:50.474127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:50.474097    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:46:50.474527 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:50.474152    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:46:53.701122 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.701090    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:46:53.740250 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.740224    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:46:53.743171 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.743152    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:46:53.757239 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757216    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-credential-socket\") pod \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") "
Oct 30 03:46:53.757348 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757244    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-kube-api-access-8f2bk\" (UniqueName: \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-kube-api-access-8f2bk\") pod \"4869afaa-7493-4149-a0b2-c3b0f4b69418\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") "
Oct 30 03:46:53.757348 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757260    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-certs\") pod \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") "
Oct 30 03:46:53.757348 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757277    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-envoy\") pod \"4869afaa-7493-4149-a0b2-c3b0f4b69418\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") "
Oct 30 03:46:53.757348 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757296    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-data\") pod \"4869afaa-7493-4149-a0b2-c3b0f4b69418\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") "
Oct 30 03:46:53.757348 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757319    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-podinfo\") pod \"4869afaa-7493-4149-a0b2-c3b0f4b69418\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") "
Oct 30 03:46:53.757348 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757334    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-certs\") pod \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") "
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757361    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-data\") pod \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") "
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757407    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-socket\") pod \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") "
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757440    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-credential-socket\") pod \"4869afaa-7493-4149-a0b2-c3b0f4b69418\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") "
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757478    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-socket\") pod \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") "
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757510    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-envoy\") pod \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") "
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757536    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4869afaa-7493-4149-a0b2-c3b0f4b69418-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4869afaa-7493-4149-a0b2-c3b0f4b69418-istiod-ca-cert\") pod \"4869afaa-7493-4149-a0b2-c3b0f4b69418\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") "
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757550    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-certs" pod "47395de4-fb30-4ad7-bf0e-14fcd9a471bb" (UID: "47395de4-fb30-4ad7-bf0e-14fcd9a471bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757570    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istiod-ca-cert\") pod \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") "
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757567    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-credential-socket" pod "47395de4-fb30-4ad7-bf0e-14fcd9a471bb" (UID: "47395de4-fb30-4ad7-bf0e-14fcd9a471bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757613    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-envoy\") pod \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") "
Oct 30 03:46:53.757650 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757646    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istiod-ca-cert\") pod \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") "
Oct 30 03:46:53.758220 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757678    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-certs\") pod \"4869afaa-7493-4149-a0b2-c3b0f4b69418\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") "
Oct 30 03:46:53.758220 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757717    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-podinfo\") pod \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") "
Oct 30 03:46:53.758220 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757743    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-token\" (UniqueName: \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-token\") pod \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") "
Oct 30 03:46:53.758220 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757786    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-token\" (UniqueName: \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-token\") pod \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") "
Oct 30 03:46:53.758220 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757644    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-data" pod "4869afaa-7493-4149-a0b2-c3b0f4b69418" (UID: "4869afaa-7493-4149-a0b2-c3b0f4b69418"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.758220 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757654    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-data" pod "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" (UID: "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.758220 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757773    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-socket" pod "47395de4-fb30-4ad7-bf0e-14fcd9a471bb" (UID: "47395de4-fb30-4ad7-bf0e-14fcd9a471bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.758220 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757785    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-certs" pod "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" (UID: "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.758220 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.758142    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-credential-socket" pod "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" (UID: "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.760716 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.758146    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4869afaa-7493-4149-a0b2-c3b0f4b69418-istiod-ca-cert" pod "4869afaa-7493-4149-a0b2-c3b0f4b69418" (UID: "4869afaa-7493-4149-a0b2-c3b0f4b69418"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:53.760849 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.758194    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istiod-ca-cert" pod "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" (UID: "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:53.761018 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.758388    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-socket" pod "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" (UID: "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.761116 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.758394    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-certs" pod "4869afaa-7493-4149-a0b2-c3b0f4b69418" (UID: "4869afaa-7493-4149-a0b2-c3b0f4b69418"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.761206 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.758629    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-credential-socket" pod "4869afaa-7493-4149-a0b2-c3b0f4b69418" (UID: "4869afaa-7493-4149-a0b2-c3b0f4b69418"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.761288 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.758882    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istiod-ca-cert" pod "47395de4-fb30-4ad7-bf0e-14fcd9a471bb" (UID: "47395de4-fb30-4ad7-bf0e-14fcd9a471bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:53.761368 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.760892    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-envoy" pod "4869afaa-7493-4149-a0b2-c3b0f4b69418" (UID: "4869afaa-7493-4149-a0b2-c3b0f4b69418"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.761818 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.761793    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-envoy" pod "47395de4-fb30-4ad7-bf0e-14fcd9a471bb" (UID: "47395de4-fb30-4ad7-bf0e-14fcd9a471bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.761818 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.757817    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-credential-socket\") pod \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") "
Oct 30 03:46:53.761991 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.761861    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-kube-api-access-rvx57\" (UniqueName: \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-kube-api-access-rvx57\") pod \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") "
Oct 30 03:46:53.761991 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.761919    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-data\") pod \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\" (UID: \"47395de4-fb30-4ad7-bf0e-14fcd9a471bb\") "
Oct 30 03:46:53.761991 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.761950    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-socket\") pod \"4869afaa-7493-4149-a0b2-c3b0f4b69418\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") "
Oct 30 03:46:53.761991 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.761986    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-kube-api-access-48hws\" (UniqueName: \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-kube-api-access-48hws\") pod \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") "
Oct 30 03:46:53.762308 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762018    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-podinfo\") pod \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\" (UID: \"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa\") "
Oct 30 03:46:53.762308 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762064    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-token\" (UniqueName: \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-token\") pod \"4869afaa-7493-4149-a0b2-c3b0f4b69418\" (UID: \"4869afaa-7493-4149-a0b2-c3b0f4b69418\") "
Oct 30 03:46:53.762308 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762070    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-envoy" pod "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" (UID: "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.762460 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762315    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-socket" pod "4869afaa-7493-4149-a0b2-c3b0f4b69418" (UID: "4869afaa-7493-4149-a0b2-c3b0f4b69418"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.762517 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762475    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762573 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762515    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762573 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762530    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762573 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762545    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762573 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762558    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4869afaa-7493-4149-a0b2-c3b0f4b69418-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762573 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762570    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762582    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762595    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762607    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762619    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762622    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-podinfo" pod "47395de4-fb30-4ad7-bf0e-14fcd9a471bb" (UID: "47395de4-fb30-4ad7-bf0e-14fcd9a471bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762631    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762659    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762670    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762679    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762686    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762697    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762709    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.762827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762711    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-podinfo" pod "4869afaa-7493-4149-a0b2-c3b0f4b69418" (UID: "4869afaa-7493-4149-a0b2-c3b0f4b69418"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:46:53.763536 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.762877    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-data" pod "47395de4-fb30-4ad7-bf0e-14fcd9a471bb" (UID: "47395de4-fb30-4ad7-bf0e-14fcd9a471bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:53.763710 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.763682    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-kube-api-access-8f2bk" pod "4869afaa-7493-4149-a0b2-c3b0f4b69418" (UID: "4869afaa-7493-4149-a0b2-c3b0f4b69418"). InnerVolumeSpecName "kube-api-access-8f2bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:53.763964 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.763718    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-token" pod "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" (UID: "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:53.764693 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.764671    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-token" pod "4869afaa-7493-4149-a0b2-c3b0f4b69418" (UID: "4869afaa-7493-4149-a0b2-c3b0f4b69418"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:53.765179 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.765149    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-kube-api-access-48hws" pod "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" (UID: "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa"). InnerVolumeSpecName "kube-api-access-48hws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:53.765274 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.765253    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-kube-api-access-rvx57" pod "47395de4-fb30-4ad7-bf0e-14fcd9a471bb" (UID: "47395de4-fb30-4ad7-bf0e-14fcd9a471bb"). InnerVolumeSpecName "kube-api-access-rvx57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:53.765653 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.765631    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-podinfo" pod "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" (UID: "ffd73cd4-c05f-441d-b9ba-556eeb50c1aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:46:53.765785 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.765771    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-token" pod "47395de4-fb30-4ad7-bf0e-14fcd9a471bb" (UID: "47395de4-fb30-4ad7-bf0e-14fcd9a471bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:53.863812 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.863778    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48hws\" (UniqueName: \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-kube-api-access-48hws\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.863983 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.863818    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.863983 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.863834    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.863983 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.863848    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f2bk\" (UniqueName: \"kubernetes.io/projected/4869afaa-7493-4149-a0b2-c3b0f4b69418-kube-api-access-8f2bk\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.863983 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.863861    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4869afaa-7493-4149-a0b2-c3b0f4b69418-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.863983 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.863875    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.863983 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.863888    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.863983 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.863929    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.863983 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.863943    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvx57\" (UniqueName: \"kubernetes.io/projected/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-kube-api-access-rvx57\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:53.863983 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:53.863956    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47395de4-fb30-4ad7-bf0e-14fcd9a471bb-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:46:54.060608 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.060569    4494 generic.go:356] "Generic (PLEG): container finished" podID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerID="0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129" exitCode=0
Oct 30 03:46:54.060786 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.060614    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"
Oct 30 03:46:54.060786 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.060648    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" event={"ID":"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa","Type":"ContainerDied","Data":"0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129"}
Oct 30 03:46:54.060786 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.060699    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94" event={"ID":"ffd73cd4-c05f-441d-b9ba-556eeb50c1aa","Type":"ContainerDied","Data":"1c1474384f271eadda2a46f9b9886c8a0960a1a571d04244e55d949dceb77b37"}
Oct 30 03:46:54.060786 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.060727    4494 scope.go:122] "RemoveContainer" containerID="0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129"
Oct 30 03:46:54.062530 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.062505    4494 generic.go:356] "Generic (PLEG): container finished" podID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerID="ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3" exitCode=0
Oct 30 03:46:54.062658 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.062551    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"
Oct 30 03:46:54.062658 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.062650    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" event={"ID":"47395de4-fb30-4ad7-bf0e-14fcd9a471bb","Type":"ContainerDied","Data":"ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3"}
Oct 30 03:46:54.062779 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.062686    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45" event={"ID":"47395de4-fb30-4ad7-bf0e-14fcd9a471bb","Type":"ContainerDied","Data":"583b6e902eb2a3e0bb9745f179f4f491d88b1a2598b7636b699dad6657335856"}
Oct 30 03:46:54.065409 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.064509    4494 generic.go:356] "Generic (PLEG): container finished" podID="4869afaa-7493-4149-a0b2-c3b0f4b69418" containerID="971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c" exitCode=0
Oct 30 03:46:54.065409 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.064577    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" event={"ID":"4869afaa-7493-4149-a0b2-c3b0f4b69418","Type":"ContainerDied","Data":"971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c"}
Oct 30 03:46:54.065409 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.064617    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x" event={"ID":"4869afaa-7493-4149-a0b2-c3b0f4b69418","Type":"ContainerDied","Data":"e2ff873d839287fc6744613e02d79a38ee21fc58d562c955431b7bc754d31791"}
Oct 30 03:46:54.065409 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.064654    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"
Oct 30 03:46:54.070140 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.070121    4494 scope.go:122] "RemoveContainer" containerID="cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2"
Oct 30 03:46:54.078360 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.078343    4494 scope.go:122] "RemoveContainer" containerID="0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129"
Oct 30 03:46:54.078662 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:46:54.078634    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129\": container with ID starting with 0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129 not found: ID does not exist" containerID="0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129"
Oct 30 03:46:54.078778 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.078665    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129"} err="failed to get container status \"0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129\": rpc error: code = NotFound desc = could not find container \"0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129\": container with ID starting with 0835c9e9e8615f0c44bb2ed8dcbdf5ccfcceef22e6cb1e2932823a825b31b129 not found: ID does not exist"
Oct 30 03:46:54.078778 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.078683    4494 scope.go:122] "RemoveContainer" containerID="cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2"
Oct 30 03:46:54.078980 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:46:54.078960    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2\": container with ID starting with cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2 not found: ID does not exist" containerID="cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2"
Oct 30 03:46:54.079072 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.078985    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2"} err="failed to get container status \"cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2\": rpc error: code = NotFound desc = could not find container \"cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2\": container with ID starting with cbbe2dfb8d97ddbb8ac401192427838d8b0e5d01cfdde8a289ffbe9d1168a9e2 not found: ID does not exist"
Oct 30 03:46:54.079072 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.079022    4494 scope.go:122] "RemoveContainer" containerID="ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3"
Oct 30 03:46:54.084496 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:46:54.084470    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:46:54.087519 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.087504    4494 scope.go:122] "RemoveContainer" containerID="3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef"
Oct 30 03:46:54.095156 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.095140    4494 scope.go:122] "RemoveContainer" containerID="ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3"
Oct 30 03:46:54.095405 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:46:54.095387    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3\": container with ID starting with ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3 not found: ID does not exist" containerID="ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3"
Oct 30 03:46:54.095456 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.095411    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3"} err="failed to get container status \"ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3\": rpc error: code = NotFound desc = could not find container \"ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3\": container with ID starting with ab5bb8e1b0e895002d766245344cc722d1cdf87b6ddf00d225071900ef03eeb3 not found: ID does not exist"
Oct 30 03:46:54.095456 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.095427    4494 scope.go:122] "RemoveContainer" containerID="3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef"
Oct 30 03:46:54.095653 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:46:54.095636    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef\": container with ID starting with 3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef not found: ID does not exist" containerID="3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef"
Oct 30 03:46:54.095712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.095668    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef"} err="failed to get container status \"3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef\": rpc error: code = NotFound desc = could not find container \"3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef\": container with ID starting with 3b73dcdd5e87dccb0c702754dbf00e88754abe4f42c7809efead5b4cc6f98bef not found: ID does not exist"
Oct 30 03:46:54.095712 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.095683    4494 scope.go:122] "RemoveContainer" containerID="971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c"
Oct 30 03:46:54.105393 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.105374    4494 scope.go:122] "RemoveContainer" containerID="971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c"
Oct 30 03:46:54.105867 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:46:54.105848    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c\": container with ID starting with 971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c not found: ID does not exist" containerID="971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c"
Oct 30 03:46:54.105984 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.105873    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c"} err="failed to get container status \"971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c\": rpc error: code = NotFound desc = could not find container \"971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c\": container with ID starting with 971f538337501d3869fb4fd989417d4c59c80b383b635f000360b7fe9d0fa23c not found: ID does not exist"
Oct 30 03:46:54.112958 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.112937    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"]
Oct 30 03:46:54.116714 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.116695    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-dnb94"]
Oct 30 03:46:54.152292 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.152267    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"]
Oct 30 03:46:54.156351 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.156330    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-wrt4x"]
Oct 30 03:46:54.189971 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.189952    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"]
Oct 30 03:46:54.193832 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:54.193813    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-lrl45"]
Oct 30 03:46:55.925940 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:55.925887    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" path="/var/lib/kubelet/pods/47395de4-fb30-4ad7-bf0e-14fcd9a471bb/volumes"
Oct 30 03:46:55.926488 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:55.926468    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4869afaa-7493-4149-a0b2-c3b0f4b69418" path="/var/lib/kubelet/pods/4869afaa-7493-4149-a0b2-c3b0f4b69418/volumes"
Oct 30 03:46:55.926892 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:46:55.926875    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" path="/var/lib/kubelet/pods/ffd73cd4-c05f-441d-b9ba-556eeb50c1aa/volumes"
Oct 30 03:47:24.082132 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:47:24.082039    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:47:54.081286 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:47:54.081248    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:48:11.563807 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.563768    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"]
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.564851    4494 cpu_manager.go:414] "containerMap: removing container" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.564877    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.564915    4494 cpu_manager.go:414] "containerMap: removing container" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.564925    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.564959    4494 cpu_manager.go:414] "containerMap: removing container" podUID="69e6755d-a445-47c7-be8a-31e668a8f3fa" containerName="main"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.564967    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e6755d-a445-47c7-be8a-31e668a8f3fa" containerName="main"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.564987    4494 cpu_manager.go:414] "containerMap: removing container" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.564995    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565014    4494 cpu_manager.go:414] "containerMap: removing container" podUID="4869afaa-7493-4149-a0b2-c3b0f4b69418" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565022    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4869afaa-7493-4149-a0b2-c3b0f4b69418" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565045    4494 cpu_manager.go:414] "containerMap: removing container" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565052    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565085    4494 cpu_manager.go:414] "containerMap: removing container" podUID="28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" containerName="main"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565093    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" containerName="main"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565345    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565370    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e7cd1e-3a7c-46c3-8c31-5d1d361e43f8" containerName="main"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565391    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565400    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e6755d-a445-47c7-be8a-31e668a8f3fa" containerName="main"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.565410    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="4869afaa-7493-4149-a0b2-c3b0f4b69418" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.566003    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd73cd4-c05f-441d-b9ba-556eeb50c1aa" containerName="istio-proxy"
Oct 30 03:48:11.566399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.566024    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="47395de4-fb30-4ad7-bf0e-14fcd9a471bb" containerName="istio-proxy"
Oct 30 03:48:11.570889 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.570856    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.586325 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.586301    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"]
Oct 30 03:48:11.596640 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.596615    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-nksxv\""
Oct 30 03:48:11.596792 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.596771    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Oct 30 03:48:11.596844 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.596792    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Oct 30 03:48:11.596844 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.596771    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Oct 30 03:48:11.646016 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.645977    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.646199 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.646037    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.646199 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.646123    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67gqf\" (UniqueName: \"kubernetes.io/projected/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kube-api-access-67gqf\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.646199 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.646153    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.646199 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.646184    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.646412 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.646315    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.646412 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.646373    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.747323 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.747289    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.747526 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.747337    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.747526 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.747509    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.747652 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.747574    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67gqf\" (UniqueName: \"kubernetes.io/projected/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kube-api-access-67gqf\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.747652 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.747615    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.747767 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.747664    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.747767 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.747694    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.747767 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.747750    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.747944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.747857    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.748056 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.748038    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.748110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.748092    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.749953 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.749934    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.750031 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.749993    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.773597 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.773561    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67gqf\" (UniqueName: \"kubernetes.io/projected/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kube-api-access-67gqf\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:11.889338 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:11.889240    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:12.283285 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:12.283253    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"]
Oct 30 03:48:12.322307 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:12.322285    4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:48:12.408546 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:12.408506    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" event={"ID":"5d26b712-5e89-445a-9fa3-9c6e44a325c1","Type":"ContainerStarted","Data":"94eeb767f7f3e57588eee026fa91c0ff086f9b00123281b8e607b3605483b33f"}
Oct 30 03:48:19.443446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:19.443406    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" event={"ID":"5d26b712-5e89-445a-9fa3-9c6e44a325c1","Type":"ContainerStarted","Data":"0bb0818212265601fd3edc5885a14c759ff046ccc79fb0a8d04b5306eb80149d"}
Oct 30 03:48:20.449827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:20.449787    4494 generic.go:356] "Generic (PLEG): container finished" podID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerID="0bb0818212265601fd3edc5885a14c759ff046ccc79fb0a8d04b5306eb80149d" exitCode=0
Oct 30 03:48:20.450280 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:20.449879    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" event={"ID":"5d26b712-5e89-445a-9fa3-9c6e44a325c1","Type":"ContainerDied","Data":"0bb0818212265601fd3edc5885a14c759ff046ccc79fb0a8d04b5306eb80149d"}
Oct 30 03:48:21.457116 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:21.457030    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" event={"ID":"5d26b712-5e89-445a-9fa3-9c6e44a325c1","Type":"ContainerStarted","Data":"aa5d3357aa4bb623d87b887c416995fcc400a36afcd6ebd223054a26443e7712"}
Oct 30 03:48:21.457116 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:21.457087    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" event={"ID":"5d26b712-5e89-445a-9fa3-9c6e44a325c1","Type":"ContainerStarted","Data":"6545e1a7620eeb4bcfcd0e98399d3c5461e27c06c144d3bf95b0b6ce0070434f"}
Oct 30 03:48:21.528869 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:21.528799    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podStartSLOduration=2.842895736 podStartE2EDuration="10.528780926s" totalImagesPullingTime="6.404327215s" totalInitContainerRuntime="1.281557975s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:21.527289676 +0000 UTC m=+3165.733469895" watchObservedRunningTime="2026-10-30 03:48:21.528780926 +0000 UTC m=+3165.734961143"
Oct 30 03:48:21.889757 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:21.889702    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:21.889994 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:21.889784    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:48:21.891238 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:21.891209    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:48:24.083346 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:48:24.083303    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:48:31.890084 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:31.890036    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:48:33.763986 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.763942    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"]
Oct 30 03:48:33.771783 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.771742    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.778124 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.778091    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"]
Oct 30 03:48:33.781975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.781950    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"]
Oct 30 03:48:33.782120 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.782076    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.785633 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.785614    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.798359 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.798330    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"]
Oct 30 03:48:33.814205 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.814164    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"]
Oct 30 03:48:33.820648 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.820614    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"]
Oct 30 03:48:33.885529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885473    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.885529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885516    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.885529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885536    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.885865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885619    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc28w\" (UniqueName: \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-kube-api-access-rc28w\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.885865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885661    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.885865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885721    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.885865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885741    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97fvg\" (UniqueName: \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-kube-api-access-97fvg\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.885865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885769    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.885865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885839    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.885865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885862    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885883    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885942    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885967    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.885997    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886051    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886070    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886085    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886099    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886180    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886203    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886220    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eca957c-c200-4e82-aab8-ad0806610ed3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.886296 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886244    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.886976 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886305    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.886976 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886332    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.886976 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886352    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.886976 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886378    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkn9k\" (UniqueName: \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-kube-api-access-gkn9k\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.886976 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.886445    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.987507 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987464    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.987711 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987518    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.987711 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987539    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.987711 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987564    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.987711 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987593    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.987711 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987653    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.987711 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987683    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987710    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987736    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987759    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987790    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987815    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987839    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eca957c-c200-4e82-aab8-ad0806610ed3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987877    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987942    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.987970    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988001    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988003    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988006    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.988118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988062    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.988790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988353    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.988790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988373    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.988790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988425    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.988790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988570    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.988790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988575    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.988790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988612    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkn9k\" (UniqueName: \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-kube-api-access-gkn9k\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.988790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988701    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.988790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988728    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.988790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988758    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.988790 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988790    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.989362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988822    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc28w\" (UniqueName: \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-kube-api-access-rc28w\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.989362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988852    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.989362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988889    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.989362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988938    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97fvg\" (UniqueName: \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-kube-api-access-97fvg\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.989362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.988975    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.989362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.989068    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.989362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.989181    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eca957c-c200-4e82-aab8-ad0806610ed3-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.989812 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.989786    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.989915 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.989880    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.989992 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.989966    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.990497 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.990125    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.990497 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.990180    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.991167 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.991111    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.991749 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.991726    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.991873 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.991844    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:33.992689 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.992670    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:33.992773 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.992714    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:33.992773 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:33.992712    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:34.013850 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.013808    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkn9k\" (UniqueName: \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-kube-api-access-gkn9k\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:34.014108 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.013943    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:34.014188 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.014151    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc28w\" (UniqueName: \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-kube-api-access-rc28w\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:34.014283 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.014263    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97fvg\" (UniqueName: \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-kube-api-access-97fvg\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:34.014337 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.014280    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-85p64\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:34.014491 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.014475    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:34.088344 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.088311    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:34.100148 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.100114    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:34.107438 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.107402    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:34.420105 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.419984    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"]
Oct 30 03:48:34.422629 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.422499    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"]
Oct 30 03:48:34.423756 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.423719    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:48:34.423843 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.423807    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:48:34.423883 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.423845    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:48:34.424241 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:48:34.424217    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded43124b_6611_49bd_ba1b_48e73fa3c14b.slice/crio-4d50ff8924de28ca2c298add2674a0ac858ea4b75b5303fa511df57625bfff36 WatchSource:0}: Error finding container 4d50ff8924de28ca2c298add2674a0ac858ea4b75b5303fa511df57625bfff36: Status 404 returned error can't find the container with id 4d50ff8924de28ca2c298add2674a0ac858ea4b75b5303fa511df57625bfff36
Oct 30 03:48:34.426445 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.426417    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:48:34.426528 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.426482    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:48:34.426528 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.426512    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:48:34.434309 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.434280    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"]
Oct 30 03:48:34.434638 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:48:34.434601    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1bf035_bfcd_4d55_b1b5_b2054d8930e7.slice/crio-2715b052a868713ab837df340c0bcccf58dfefb6898c037a4af0fd2d80bac898 WatchSource:0}: Error finding container 2715b052a868713ab837df340c0bcccf58dfefb6898c037a4af0fd2d80bac898: Status 404 returned error can't find the container with id 2715b052a868713ab837df340c0bcccf58dfefb6898c037a4af0fd2d80bac898
Oct 30 03:48:34.440416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.440384    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:48:34.440527 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.440470    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:48:34.440527 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.440515    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:48:34.529123 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.522956    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" event={"ID":"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7","Type":"ContainerStarted","Data":"5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7"}
Oct 30 03:48:34.529123 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.523012    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" event={"ID":"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7","Type":"ContainerStarted","Data":"2715b052a868713ab837df340c0bcccf58dfefb6898c037a4af0fd2d80bac898"}
Oct 30 03:48:34.530391 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.530272    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" event={"ID":"ed43124b-6611-49bd-ba1b-48e73fa3c14b","Type":"ContainerStarted","Data":"c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389"}
Oct 30 03:48:34.530391 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.530345    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" event={"ID":"ed43124b-6611-49bd-ba1b-48e73fa3c14b","Type":"ContainerStarted","Data":"4d50ff8924de28ca2c298add2674a0ac858ea4b75b5303fa511df57625bfff36"}
Oct 30 03:48:34.533002 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.532729    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" event={"ID":"2eca957c-c200-4e82-aab8-ad0806610ed3","Type":"ContainerStarted","Data":"a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68"}
Oct 30 03:48:34.533002 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.532803    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" event={"ID":"2eca957c-c200-4e82-aab8-ad0806610ed3","Type":"ContainerStarted","Data":"db33ebbfad40402f325d34204d7a4a0dc45e7ff18f4d2cefdc420bd2579bbe63"}
Oct 30 03:48:34.617434 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.617361    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" podStartSLOduration=1.61733824 podStartE2EDuration="1.61733824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:34.571371997 +0000 UTC m=+3178.777552216" watchObservedRunningTime="2026-10-30 03:48:34.61733824 +0000 UTC m=+3178.823518459"
Oct 30 03:48:34.618123 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.618066    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" podStartSLOduration=1.618054539 podStartE2EDuration="1.618054539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:34.616449419 +0000 UTC m=+3178.822629647" watchObservedRunningTime="2026-10-30 03:48:34.618054539 +0000 UTC m=+3178.824234756"
Oct 30 03:48:34.667546 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:34.667414    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" podStartSLOduration=1.6673923419999999 podStartE2EDuration="1.667392342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:34.666957607 +0000 UTC m=+3178.873137843" watchObservedRunningTime="2026-10-30 03:48:34.667392342 +0000 UTC m=+3178.873572560"
Oct 30 03:48:35.091770 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:35.091733    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:35.092493 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:35.092462    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:48:35.092644 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:35.092619    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 03:48:35.102526 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:35.102492    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:35.102836 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:35.102816    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 03:48:35.103047 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:35.103017    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 03:48:35.109031 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:35.108997    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:35.109362 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:35.109335    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 03:48:35.109532 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:35.109508    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 03:48:36.090925 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:36.089374    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:48:36.090925 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:36.089441    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 03:48:36.104600 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:36.104541    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 03:48:36.105276 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:36.105220    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 03:48:36.109364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:36.109334    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 03:48:36.109606 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:36.109568    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 03:48:37.090928 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:37.089399    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:48:37.090928 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:37.089475    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 03:48:37.102927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:37.102454    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 03:48:37.102927 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:37.102522    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 03:48:37.120929 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:37.117751    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 03:48:37.120929 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:37.117817    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 03:48:38.090925 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:38.089459    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:48:38.090925 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:38.089546    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 03:48:38.104926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:38.103350    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 03:48:38.104926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:38.103415    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 03:48:38.118926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:38.117073    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 03:48:38.118926 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:38.117199    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 03:48:39.089937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:39.089365    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:48:39.089937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:39.089430    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 03:48:39.107941 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:39.107328    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 03:48:39.107941 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:39.107390    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 03:48:39.108226 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:39.108202    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 03:48:39.108290 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:39.108250    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 03:48:40.089836 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:40.089379    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 03:48:40.089836 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:40.089444    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 03:48:40.101483 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:40.101355    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 03:48:40.101483 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:40.101421    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 03:48:40.110081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:40.110032    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 03:48:40.110367 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:40.110336    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 03:48:41.093412 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:41.093377    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:41.093935 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:41.093668    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:41.094433 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:41.094412    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:48:41.104719 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:41.104698    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:41.104963 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:41.104945    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:41.105667 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:41.105652    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:48:41.111891 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:41.111862    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:41.112300 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:41.112277    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:41.113067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:41.113046    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:48:41.890330 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:41.890272    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:48:51.890576 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:48:51.890473    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:48:54.082893 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:48:54.082844    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:49:01.890473 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:49:01.890419    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:49:11.890486 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:49:11.890438    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:49:21.889847 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:49:21.889740    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:49:24.084423 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:49:24.084369    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:49:31.890147 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:49:31.890097    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:49:41.890322 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:49:41.890271    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:49:51.889863 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:49:51.889813    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:49:54.084630 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:49:54.084589    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:50:01.890487 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:01.890442    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:50:11.890436 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:11.890390    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:50:21.900768 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:21.900682    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:50:21.909113 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:21.909079    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:50:24.082692 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:50:24.082655    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:50:27.322088 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:27.322058    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"]
Oct 30 03:50:27.322549 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:27.322364    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="modelcar" containerID="cri-o://aa5d3357aa4bb623d87b887c416995fcc400a36afcd6ebd223054a26443e7712" gracePeriod=60
Oct 30 03:50:28.129344 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:28.129314    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q_5d26b712-5e89-445a-9fa3-9c6e44a325c1/modelcar/0.log"
Oct 30 03:50:28.130014 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:28.129988    4494 generic.go:356] "Generic (PLEG): container finished" podID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerID="aa5d3357aa4bb623d87b887c416995fcc400a36afcd6ebd223054a26443e7712" exitCode=143
Oct 30 03:50:28.130095 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:28.130057    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" event={"ID":"5d26b712-5e89-445a-9fa3-9c6e44a325c1","Type":"ContainerDied","Data":"aa5d3357aa4bb623d87b887c416995fcc400a36afcd6ebd223054a26443e7712"}
Oct 30 03:50:36.228155 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:36.228126    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q_5d26b712-5e89-445a-9fa3-9c6e44a325c1/modelcar/0.log"
Oct 30 03:50:36.237740 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:36.237716    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q_5d26b712-5e89-445a-9fa3-9c6e44a325c1/modelcar/0.log"
Oct 30 03:50:42.345966 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:42.345886    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" containerID="cri-o://6545e1a7620eeb4bcfcd0e98399d3c5461e27c06c144d3bf95b0b6ce0070434f" gracePeriod=45
Oct 30 03:50:42.901933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:42.901872    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Oct 30 03:50:43.198061 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.197976    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q_5d26b712-5e89-445a-9fa3-9c6e44a325c1/modelcar/0.log"
Oct 30 03:50:43.198679 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.198657    4494 generic.go:356] "Generic (PLEG): container finished" podID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerID="6545e1a7620eeb4bcfcd0e98399d3c5461e27c06c144d3bf95b0b6ce0070434f" exitCode=0
Oct 30 03:50:43.198750 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.198732    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" event={"ID":"5d26b712-5e89-445a-9fa3-9c6e44a325c1","Type":"ContainerDied","Data":"6545e1a7620eeb4bcfcd0e98399d3c5461e27c06c144d3bf95b0b6ce0070434f"}
Oct 30 03:50:43.607173 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.607147    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q_5d26b712-5e89-445a-9fa3-9c6e44a325c1/modelcar/0.log"
Oct 30 03:50:43.610016 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.609998    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:50:43.687201 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.687168    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tmp-dir\") pod \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") "
Oct 30 03:50:43.687386 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.687209    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-home\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-home\") pod \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") "
Oct 30 03:50:43.687386 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.687235    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-model-cache\") pod \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") "
Oct 30 03:50:43.687386 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.687259    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-dshm\") pod \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") "
Oct 30 03:50:43.687386 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.687301    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tls-certs\") pod \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") "
Oct 30 03:50:43.687386 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.687359    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kube-api-access-67gqf\" (UniqueName: \"kubernetes.io/projected/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kube-api-access-67gqf\") pod \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") "
Oct 30 03:50:43.687645 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.687626    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kserve-provision-location\") pod \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\" (UID: \"5d26b712-5e89-445a-9fa3-9c6e44a325c1\") "
Oct 30 03:50:43.688038 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.687889    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kserve-provision-location" pod "5d26b712-5e89-445a-9fa3-9c6e44a325c1" (UID: "5d26b712-5e89-445a-9fa3-9c6e44a325c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:43.688038 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.688012    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:50:43.688237 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.688142    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-model-cache" pod "5d26b712-5e89-445a-9fa3-9c6e44a325c1" (UID: "5d26b712-5e89-445a-9fa3-9c6e44a325c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:43.688237 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.688218    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-home" pod "5d26b712-5e89-445a-9fa3-9c6e44a325c1" (UID: "5d26b712-5e89-445a-9fa3-9c6e44a325c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:43.690468 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.690439    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-dshm" pod "5d26b712-5e89-445a-9fa3-9c6e44a325c1" (UID: "5d26b712-5e89-445a-9fa3-9c6e44a325c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:43.690593 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.690501    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tls-certs" pod "5d26b712-5e89-445a-9fa3-9c6e44a325c1" (UID: "5d26b712-5e89-445a-9fa3-9c6e44a325c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:43.690593 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.690510    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kube-api-access-67gqf" pod "5d26b712-5e89-445a-9fa3-9c6e44a325c1" (UID: "5d26b712-5e89-445a-9fa3-9c6e44a325c1"). InnerVolumeSpecName "kube-api-access-67gqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:43.699495 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.699471    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tmp-dir" pod "5d26b712-5e89-445a-9fa3-9c6e44a325c1" (UID: "5d26b712-5e89-445a-9fa3-9c6e44a325c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:43.789301 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.789259    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:50:43.789301 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.789296    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:50:43.789301 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.789307    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:50:43.789542 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.789316    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d26b712-5e89-445a-9fa3-9c6e44a325c1-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:50:43.789542 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.789324    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d26b712-5e89-445a-9fa3-9c6e44a325c1-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:50:43.789542 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:43.789334    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67gqf\" (UniqueName: \"kubernetes.io/projected/5d26b712-5e89-445a-9fa3-9c6e44a325c1-kube-api-access-67gqf\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:50:44.204348 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:44.204256    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q_5d26b712-5e89-445a-9fa3-9c6e44a325c1/modelcar/0.log"
Oct 30 03:50:44.205244 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:44.205222    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"
Oct 30 03:50:44.205385 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:44.205241    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q" event={"ID":"5d26b712-5e89-445a-9fa3-9c6e44a325c1","Type":"ContainerDied","Data":"94eeb767f7f3e57588eee026fa91c0ff086f9b00123281b8e607b3605483b33f"}
Oct 30 03:50:44.205385 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:44.205318    4494 scope.go:122] "RemoveContainer" containerID="aa5d3357aa4bb623d87b887c416995fcc400a36afcd6ebd223054a26443e7712"
Oct 30 03:50:44.216311 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:44.216292    4494 scope.go:122] "RemoveContainer" containerID="6545e1a7620eeb4bcfcd0e98399d3c5461e27c06c144d3bf95b0b6ce0070434f"
Oct 30 03:50:44.224332 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:44.224310    4494 scope.go:122] "RemoveContainer" containerID="0bb0818212265601fd3edc5885a14c759ff046ccc79fb0a8d04b5306eb80149d"
Oct 30 03:50:44.257955 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:44.257931    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"]
Oct 30 03:50:44.270815 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:44.270782    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9brnw7q"]
Oct 30 03:50:45.926829 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:50:45.926793    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" path="/var/lib/kubelet/pods/5d26b712-5e89-445a-9fa3-9c6e44a325c1/volumes"
Oct 30 03:50:54.081583 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:50:54.081547    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:51:07.706504 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.706472    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"]
Oct 30 03:51:07.707081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.706839    4494 cpu_manager.go:414] "containerMap: removing container" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="modelcar"
Oct 30 03:51:07.707081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.706851    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="modelcar"
Oct 30 03:51:07.707081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.706873    4494 cpu_manager.go:414] "containerMap: removing container" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="modelcar-init"
Oct 30 03:51:07.707081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.706878    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="modelcar-init"
Oct 30 03:51:07.707081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.706893    4494 cpu_manager.go:414] "containerMap: removing container" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main"
Oct 30 03:51:07.707081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.706916    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main"
Oct 30 03:51:07.707081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.706988    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="main"
Oct 30 03:51:07.707081 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.706997    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d26b712-5e89-445a-9fa3-9c6e44a325c1" containerName="modelcar"
Oct 30 03:51:07.712187 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.712164    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.725852 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.725822    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"]
Oct 30 03:51:07.736381 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.736321    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:51:07.736381 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.736320    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Oct 30 03:51:07.736381 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.736320    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Oct 30 03:51:07.736700 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.736355    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-7cnqp\""
Oct 30 03:51:07.736700 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.736422    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Oct 30 03:51:07.736700 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.736363    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Oct 30 03:51:07.813519 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.813479    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe96f39-f9b7-4c50-8d42-e041fb998571-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.813519 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.813521    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg74n\" (UniqueName: \"kubernetes.io/projected/ffe96f39-f9b7-4c50-8d42-e041fb998571-kube-api-access-sg74n\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.813780 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.813574    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.813780 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.813610    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.813780 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.813677    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.813780 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.813701    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.813780 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.813728    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.813979 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.813796    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffe96f39-f9b7-4c50-8d42-e041fb998571-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915161 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915124    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe96f39-f9b7-4c50-8d42-e041fb998571-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915161 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915160    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg74n\" (UniqueName: \"kubernetes.io/projected/ffe96f39-f9b7-4c50-8d42-e041fb998571-kube-api-access-sg74n\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915180    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915231    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915284    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915323    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915367    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915437 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915405    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffe96f39-f9b7-4c50-8d42-e041fb998571-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915761 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915712    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915815 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915767    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915815 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915770    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.915959 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.915844    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.916191 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.916120    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffe96f39-f9b7-4c50-8d42-e041fb998571-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.917846 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.917825    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.917973 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.917887    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe96f39-f9b7-4c50-8d42-e041fb998571-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:07.941550 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:07.941514    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg74n\" (UniqueName: \"kubernetes.io/projected/ffe96f39-f9b7-4c50-8d42-e041fb998571-kube-api-access-sg74n\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:08.028713 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:08.028676    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:08.351676 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:08.351562    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"]
Oct 30 03:51:09.325770 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:09.325673    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" event={"ID":"ffe96f39-f9b7-4c50-8d42-e041fb998571","Type":"ContainerStarted","Data":"73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7"}
Oct 30 03:51:09.325770 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:09.325740    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" event={"ID":"ffe96f39-f9b7-4c50-8d42-e041fb998571","Type":"ContainerStarted","Data":"9c00de761808b1a8c100dcba4dd62e860fee0ffefa38aff8e713dbf12117fd15"}
Oct 30 03:51:11.337202 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:11.337157    4494 generic.go:356] "Generic (PLEG): container finished" podID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerID="73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7" exitCode=0
Oct 30 03:51:11.337603 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:11.337233    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" event={"ID":"ffe96f39-f9b7-4c50-8d42-e041fb998571","Type":"ContainerDied","Data":"73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7"}
Oct 30 03:51:12.344049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:12.344001    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" event={"ID":"ffe96f39-f9b7-4c50-8d42-e041fb998571","Type":"ContainerStarted","Data":"6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92"}
Oct 30 03:51:12.417753 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:12.417685    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podStartSLOduration=2.5727383440000002 podStartE2EDuration="5.417662296s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.844923952s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:12.403267494 +0000 UTC m=+3336.609447729" watchObservedRunningTime="2026-10-30 03:51:12.417662296 +0000 UTC m=+3336.623842515"
Oct 30 03:51:13.029079 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:13.029035    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:18.029550 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:18.029511    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:51:18.031108 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:18.031068    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 30 03:51:21.193235 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.193195    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"]
Oct 30 03:51:21.197176 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.197149    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.215254 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.215175    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:51:21.215254 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.215191    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Oct 30 03:51:21.215254 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.215244    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Oct 30 03:51:21.227590 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.227552    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Oct 30 03:51:21.227776 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.227633    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-b6qz6\""
Oct 30 03:51:21.227776 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.227673    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Oct 30 03:51:21.230463 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.230432    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"]
Oct 30 03:51:21.248824 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.248792    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-model-cache\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.249049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.248831    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-dshm\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.249049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.248856    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tls-certs\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.249049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.249007    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhf8\" (UniqueName: \"kubernetes.io/projected/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kube-api-access-7fhf8\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.249236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.249052    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-home\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.249236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.249138    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.249236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.249194    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tmp-dir\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.249236 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.249232    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83c73456-0715-4de7-b3c1-8a4a991dd6eb-cabundle-cert\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.350416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.350380    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-model-cache\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.350416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.350422    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-dshm\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.350698 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.350442    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tls-certs\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.350698 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.350470    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhf8\" (UniqueName: \"kubernetes.io/projected/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kube-api-access-7fhf8\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.350698 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.350493    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-home\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.350698 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.350527    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.350698 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.350578    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tmp-dir\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.350698 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.350620    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83c73456-0715-4de7-b3c1-8a4a991dd6eb-cabundle-cert\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.351049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.351005    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tmp-dir\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.351185 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.351149    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-model-cache\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.351261 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.351232    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.351347 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.351312    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-home\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.351470 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.351431    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83c73456-0715-4de7-b3c1-8a4a991dd6eb-cabundle-cert\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.353576 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.353549    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-dshm\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.353780 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.353762    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tls-certs\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.375369 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.375337    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhf8\" (UniqueName: \"kubernetes.io/projected/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kube-api-access-7fhf8\") pod \"custom-route-timeout-test-kserve-9f98557f7-k8kqq\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.511393 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.511301    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:21.835571 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:51:21.835540    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c73456_0715_4de7_b3c1_8a4a991dd6eb.slice/crio-d893ddd9a2b9d7130134308a3aa0ef0e977966da67a2659597d366fe87dc9df8 WatchSource:0}: Error finding container d893ddd9a2b9d7130134308a3aa0ef0e977966da67a2659597d366fe87dc9df8: Status 404 returned error can't find the container with id d893ddd9a2b9d7130134308a3aa0ef0e977966da67a2659597d366fe87dc9df8
Oct 30 03:51:21.836680 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:21.836652    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"]
Oct 30 03:51:22.396983 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:22.396942    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" event={"ID":"83c73456-0715-4de7-b3c1-8a4a991dd6eb","Type":"ContainerStarted","Data":"d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216"}
Oct 30 03:51:22.397484 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:22.397008    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" event={"ID":"83c73456-0715-4de7-b3c1-8a4a991dd6eb","Type":"ContainerStarted","Data":"d893ddd9a2b9d7130134308a3aa0ef0e977966da67a2659597d366fe87dc9df8"}
Oct 30 03:51:24.085412 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:51:24.085368    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:51:24.409612 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:24.409568    4494 generic.go:356] "Generic (PLEG): container finished" podID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerID="d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216" exitCode=0
Oct 30 03:51:24.409828 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:24.409644    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" event={"ID":"83c73456-0715-4de7-b3c1-8a4a991dd6eb","Type":"ContainerDied","Data":"d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216"}
Oct 30 03:51:25.417269 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:25.417228    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" event={"ID":"83c73456-0715-4de7-b3c1-8a4a991dd6eb","Type":"ContainerStarted","Data":"0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1"}
Oct 30 03:51:25.490686 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:25.490614    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podStartSLOduration=2.059607273 podStartE2EDuration="4.490593044s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.430985771s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:25.476308637 +0000 UTC m=+3349.682488857" watchObservedRunningTime="2026-10-30 03:51:25.490593044 +0000 UTC m=+3349.696773265"
Oct 30 03:51:25.512382 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:25.512345    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:28.029247 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:28.029195    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 30 03:51:31.511856 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:31.511815    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:51:31.513492 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:31.513465    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:51:38.030101 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:38.030017    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 30 03:51:41.512270 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:41.512213    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:51:48.029854 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:48.029746    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 30 03:51:51.512402 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:51.512356    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:51:54.083876 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:51:54.083833    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:51:58.029919 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:51:58.029851    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 30 03:52:01.512789 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:01.512741    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:52:08.030212 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:08.030170    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 30 03:52:11.512325 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:11.512270    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:52:18.029965 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:18.029885    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 30 03:52:21.511811 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:21.511764    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:52:24.084315 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:52:24.084271    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:52:28.029652 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:28.029600    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 30 03:52:31.512250 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:31.512205    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:52:38.045413 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:38.045383    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:52:38.057667 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:38.057631    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:52:40.544251 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:40.544204    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"]
Oct 30 03:52:41.512301 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:41.512257    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:52:51.512205 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:51.512152    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:52:54.083800 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:52:54.083760    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:52:55.583828 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:55.583781    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main" containerID="cri-o://6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92" gracePeriod=45
Oct 30 03:52:56.046387 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.046346    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 30 03:52:56.437720 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.437691    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:52:56.549647 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.549610    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-kserve-provision-location\") pod \"ffe96f39-f9b7-4c50-8d42-e041fb998571\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") "
Oct 30 03:52:56.549838 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.549677    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-dshm\") pod \"ffe96f39-f9b7-4c50-8d42-e041fb998571\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") "
Oct 30 03:52:56.549838 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.549776    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-home\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-home\") pod \"ffe96f39-f9b7-4c50-8d42-e041fb998571\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") "
Oct 30 03:52:56.549838 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.549822    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe96f39-f9b7-4c50-8d42-e041fb998571-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe96f39-f9b7-4c50-8d42-e041fb998571-tls-certs\") pod \"ffe96f39-f9b7-4c50-8d42-e041fb998571\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") "
Oct 30 03:52:56.550011 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.549868    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe96f39-f9b7-4c50-8d42-e041fb998571-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffe96f39-f9b7-4c50-8d42-e041fb998571-cabundle-cert\") pod \"ffe96f39-f9b7-4c50-8d42-e041fb998571\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") "
Oct 30 03:52:56.550011 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.549930    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-model-cache\") pod \"ffe96f39-f9b7-4c50-8d42-e041fb998571\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") "
Oct 30 03:52:56.550011 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.549970    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe96f39-f9b7-4c50-8d42-e041fb998571-kube-api-access-sg74n\" (UniqueName: \"kubernetes.io/projected/ffe96f39-f9b7-4c50-8d42-e041fb998571-kube-api-access-sg74n\") pod \"ffe96f39-f9b7-4c50-8d42-e041fb998571\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") "
Oct 30 03:52:56.550126 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.550016    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-tmp-dir\") pod \"ffe96f39-f9b7-4c50-8d42-e041fb998571\" (UID: \"ffe96f39-f9b7-4c50-8d42-e041fb998571\") "
Oct 30 03:52:56.550866 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.550249    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-home" pod "ffe96f39-f9b7-4c50-8d42-e041fb998571" (UID: "ffe96f39-f9b7-4c50-8d42-e041fb998571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:52:56.550866 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.550386    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-model-cache" pod "ffe96f39-f9b7-4c50-8d42-e041fb998571" (UID: "ffe96f39-f9b7-4c50-8d42-e041fb998571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:52:56.550866 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.550800    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-tmp-dir" pod "ffe96f39-f9b7-4c50-8d42-e041fb998571" (UID: "ffe96f39-f9b7-4c50-8d42-e041fb998571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:52:56.550866 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.550829    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe96f39-f9b7-4c50-8d42-e041fb998571-cabundle-cert" pod "ffe96f39-f9b7-4c50-8d42-e041fb998571" (UID: "ffe96f39-f9b7-4c50-8d42-e041fb998571"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:52:56.552606 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.552545    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe96f39-f9b7-4c50-8d42-e041fb998571-tls-certs" pod "ffe96f39-f9b7-4c50-8d42-e041fb998571" (UID: "ffe96f39-f9b7-4c50-8d42-e041fb998571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:52:56.552606 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.552564    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-dshm" pod "ffe96f39-f9b7-4c50-8d42-e041fb998571" (UID: "ffe96f39-f9b7-4c50-8d42-e041fb998571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:52:56.553142 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.553117    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe96f39-f9b7-4c50-8d42-e041fb998571-kube-api-access-sg74n" pod "ffe96f39-f9b7-4c50-8d42-e041fb998571" (UID: "ffe96f39-f9b7-4c50-8d42-e041fb998571"). InnerVolumeSpecName "kube-api-access-sg74n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:52:56.566653 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.566614    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-kserve-provision-location" pod "ffe96f39-f9b7-4c50-8d42-e041fb998571" (UID: "ffe96f39-f9b7-4c50-8d42-e041fb998571"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:52:56.651161 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.651126    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:52:56.651161 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.651158    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe96f39-f9b7-4c50-8d42-e041fb998571-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:52:56.651161 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.651169    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffe96f39-f9b7-4c50-8d42-e041fb998571-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:52:56.651644 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.651177    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:52:56.651644 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.651186    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg74n\" (UniqueName: \"kubernetes.io/projected/ffe96f39-f9b7-4c50-8d42-e041fb998571-kube-api-access-sg74n\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:52:56.651644 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.651194    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:52:56.651644 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.651203    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:52:56.651644 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.651210    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe96f39-f9b7-4c50-8d42-e041fb998571-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:52:56.898558 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.898438    4494 generic.go:356] "Generic (PLEG): container finished" podID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerID="6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92" exitCode=0
Oct 30 03:52:56.898558 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.898530    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" event={"ID":"ffe96f39-f9b7-4c50-8d42-e041fb998571","Type":"ContainerDied","Data":"6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92"}
Oct 30 03:52:56.898815 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.898558    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"
Oct 30 03:52:56.898815 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.898596    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv" event={"ID":"ffe96f39-f9b7-4c50-8d42-e041fb998571","Type":"ContainerDied","Data":"9c00de761808b1a8c100dcba4dd62e860fee0ffefa38aff8e713dbf12117fd15"}
Oct 30 03:52:56.898815 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.898631    4494 scope.go:122] "RemoveContainer" containerID="6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92"
Oct 30 03:52:56.911556 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.911509    4494 scope.go:122] "RemoveContainer" containerID="73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7"
Oct 30 03:52:56.923333 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.923232    4494 scope.go:122] "RemoveContainer" containerID="6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92"
Oct 30 03:52:56.924427 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:52:56.924388    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92\": container with ID starting with 6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92 not found: ID does not exist" containerID="6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92"
Oct 30 03:52:56.924547 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.924445    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92"} err="failed to get container status \"6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92\": rpc error: code = NotFound desc = could not find container \"6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92\": container with ID starting with 6808b33cfd2e8b068efcaaa5d2840fddd740846d389c29589a2432be33eb4a92 not found: ID does not exist"
Oct 30 03:52:56.924547 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.924476    4494 scope.go:122] "RemoveContainer" containerID="73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7"
Oct 30 03:52:56.924890 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:52:56.924861    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7\": container with ID starting with 73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7 not found: ID does not exist" containerID="73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7"
Oct 30 03:52:56.924999 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.924919    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7"} err="failed to get container status \"73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7\": rpc error: code = NotFound desc = could not find container \"73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7\": container with ID starting with 73b2ca6a4b3fab8a5203f338d2d361c883dd6f01ddb3f8f57fdbfd8b706ff5e7 not found: ID does not exist"
Oct 30 03:52:56.955368 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.955327    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"]
Oct 30 03:52:56.966033 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:56.965999    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-6459d9d4dc-wkctv"]
Oct 30 03:52:57.926476 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:52:57.926440    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" path="/var/lib/kubelet/pods/ffe96f39-f9b7-4c50-8d42-e041fb998571/volumes"
Oct 30 03:53:01.512421 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:01.512372    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:53:11.512783 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:11.512723    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:53:21.512458 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:21.512364    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:53:24.083582 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:53:24.083532    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:53:31.522775 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:31.522739    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:53:31.531393 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:31.531364    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:53:39.179483 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:39.179448    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"]
Oct 30 03:53:49.673999 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:49.673962    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"]
Oct 30 03:53:49.674625 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:49.674167    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerName="istio-proxy" containerID="cri-o://a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68" gracePeriod=30
Oct 30 03:53:49.686464 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:49.686435    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"]
Oct 30 03:53:49.686674 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:49.686649    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerName="istio-proxy" containerID="cri-o://5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7" gracePeriod=30
Oct 30 03:53:49.688007 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:49.687988    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"]
Oct 30 03:53:49.688167 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:49.688149    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerName="istio-proxy" containerID="cri-o://46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99" gracePeriod=30
Oct 30 03:53:49.698684 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:49.698656    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"]
Oct 30 03:53:49.698880 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:49.698859    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerName="istio-proxy" containerID="cri-o://c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389" gracePeriod=30
Oct 30 03:53:51.613331 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:51.613294    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 03:53:51.613721 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:51.613360    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:53:54.081401 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:53:54.081359    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:53:54.203510 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:54.203464    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" containerID="cri-o://0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1" gracePeriod=45
Oct 30 03:53:54.523970 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:54.523861    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 03:53:54.990286 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:54.989976    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:53:55.043849 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.043810    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:53:55.047461 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.047209    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:53:55.054623 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.054597    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:53:55.058090 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.058067    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:53:55.118074 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118045    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-kube-api-access-gkn9k\" (UniqueName: \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-kube-api-access-gkn9k\") pod \"2eca957c-c200-4e82-aab8-ad0806610ed3\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") "
Oct 30 03:53:55.118074 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118080    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-podinfo\") pod \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118131    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-socket\") pod \"2eca957c-c200-4e82-aab8-ad0806610ed3\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118208    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-envoy\") pod \"2eca957c-c200-4e82-aab8-ad0806610ed3\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118240    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-kube-api-access-97fvg\" (UniqueName: \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-kube-api-access-97fvg\") pod \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118283    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-certs\") pod \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118314    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-data\") pod \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118339    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eca957c-c200-4e82-aab8-ad0806610ed3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eca957c-c200-4e82-aab8-ad0806610ed3-istiod-ca-cert\") pod \"2eca957c-c200-4e82-aab8-ad0806610ed3\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118380    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-certs\") pod \"2eca957c-c200-4e82-aab8-ad0806610ed3\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118438    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-token\" (UniqueName: \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-token\") pod \"2eca957c-c200-4e82-aab8-ad0806610ed3\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118463    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-socket\") pod \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118498    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-envoy\") pod \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118500    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-socket" pod "2eca957c-c200-4e82-aab8-ad0806610ed3" (UID: "2eca957c-c200-4e82-aab8-ad0806610ed3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118543    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-token\" (UniqueName: \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-token\") pod \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118589    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-data\") pod \"2eca957c-c200-4e82-aab8-ad0806610ed3\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") "
Oct 30 03:53:55.118605 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118593    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-certs" pod "ed43124b-6611-49bd-ba1b-48e73fa3c14b" (UID: "ed43124b-6611-49bd-ba1b-48e73fa3c14b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118606    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-data" pod "ed43124b-6611-49bd-ba1b-48e73fa3c14b" (UID: "ed43124b-6611-49bd-ba1b-48e73fa3c14b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118628    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-credential-socket\") pod \"2eca957c-c200-4e82-aab8-ad0806610ed3\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") "
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118690    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istiod-ca-cert\") pod \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") "
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118728    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-credential-socket\") pod \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\" (UID: \"ed43124b-6611-49bd-ba1b-48e73fa3c14b\") "
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118824    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-podinfo\") pod \"2eca957c-c200-4e82-aab8-ad0806610ed3\" (UID: \"2eca957c-c200-4e82-aab8-ad0806610ed3\") "
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.118834    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-credential-socket" pod "2eca957c-c200-4e82-aab8-ad0806610ed3" (UID: "2eca957c-c200-4e82-aab8-ad0806610ed3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.119326    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.119347    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.119363    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.119377    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.119423    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-socket" pod "ed43124b-6611-49bd-ba1b-48e73fa3c14b" (UID: "ed43124b-6611-49bd-ba1b-48e73fa3c14b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.119446 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.119442    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-certs" pod "2eca957c-c200-4e82-aab8-ad0806610ed3" (UID: "2eca957c-c200-4e82-aab8-ad0806610ed3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.120260 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.119511    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-data" pod "2eca957c-c200-4e82-aab8-ad0806610ed3" (UID: "2eca957c-c200-4e82-aab8-ad0806610ed3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.120260 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.119732    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eca957c-c200-4e82-aab8-ad0806610ed3-istiod-ca-cert" pod "2eca957c-c200-4e82-aab8-ad0806610ed3" (UID: "2eca957c-c200-4e82-aab8-ad0806610ed3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:55.120260 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.119739    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istiod-ca-cert" pod "ed43124b-6611-49bd-ba1b-48e73fa3c14b" (UID: "ed43124b-6611-49bd-ba1b-48e73fa3c14b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:55.120260 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.119912    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-credential-socket" pod "ed43124b-6611-49bd-ba1b-48e73fa3c14b" (UID: "ed43124b-6611-49bd-ba1b-48e73fa3c14b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.121378 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.121353    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-envoy" pod "2eca957c-c200-4e82-aab8-ad0806610ed3" (UID: "2eca957c-c200-4e82-aab8-ad0806610ed3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.121850 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.121825    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-token" pod "2eca957c-c200-4e82-aab8-ad0806610ed3" (UID: "2eca957c-c200-4e82-aab8-ad0806610ed3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.121850 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.121834    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-kube-api-access-gkn9k" pod "2eca957c-c200-4e82-aab8-ad0806610ed3" (UID: "2eca957c-c200-4e82-aab8-ad0806610ed3"). InnerVolumeSpecName "kube-api-access-gkn9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.122051 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.121848    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-token" pod "ed43124b-6611-49bd-ba1b-48e73fa3c14b" (UID: "ed43124b-6611-49bd-ba1b-48e73fa3c14b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.122051 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.122014    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-kube-api-access-97fvg" pod "ed43124b-6611-49bd-ba1b-48e73fa3c14b" (UID: "ed43124b-6611-49bd-ba1b-48e73fa3c14b"). InnerVolumeSpecName "kube-api-access-97fvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.122141 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.122123    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-podinfo" pod "2eca957c-c200-4e82-aab8-ad0806610ed3" (UID: "2eca957c-c200-4e82-aab8-ad0806610ed3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:53:55.122292 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.122279    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-envoy" pod "ed43124b-6611-49bd-ba1b-48e73fa3c14b" (UID: "ed43124b-6611-49bd-ba1b-48e73fa3c14b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.122536 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.122523    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-podinfo" pod "ed43124b-6611-49bd-ba1b-48e73fa3c14b" (UID: "ed43124b-6611-49bd-ba1b-48e73fa3c14b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:53:55.182196 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.182158    4494 generic.go:356] "Generic (PLEG): container finished" podID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerID="a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68" exitCode=0
Oct 30 03:53:55.182364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.182213    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"
Oct 30 03:53:55.182364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.182240    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" event={"ID":"2eca957c-c200-4e82-aab8-ad0806610ed3","Type":"ContainerDied","Data":"a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68"}
Oct 30 03:53:55.182364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.182285    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64" event={"ID":"2eca957c-c200-4e82-aab8-ad0806610ed3","Type":"ContainerDied","Data":"db33ebbfad40402f325d34204d7a4a0dc45e7ff18f4d2cefdc420bd2579bbe63"}
Oct 30 03:53:55.182364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.182306    4494 scope.go:122] "RemoveContainer" containerID="a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68"
Oct 30 03:53:55.183575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.183543    4494 generic.go:356] "Generic (PLEG): container finished" podID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerID="46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99" exitCode=0
Oct 30 03:53:55.183575 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.183572    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" event={"ID":"51422a6f-9df9-4283-815e-a0dd6660cb3f","Type":"ContainerDied","Data":"46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99"}
Oct 30 03:53:55.183772 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.183605    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"
Oct 30 03:53:55.183772 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.183615    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65" event={"ID":"51422a6f-9df9-4283-815e-a0dd6660cb3f","Type":"ContainerDied","Data":"b7c1d1c4b1810c10ce103b894c9c3b1a0f8d97d2237f390d1265f634e4809473"}
Oct 30 03:53:55.185852 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.185462    4494 generic.go:356] "Generic (PLEG): container finished" podID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerID="5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7" exitCode=0
Oct 30 03:53:55.185852 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.185523    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"
Oct 30 03:53:55.185852 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.185541    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" event={"ID":"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7","Type":"ContainerDied","Data":"5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7"}
Oct 30 03:53:55.185852 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.185575    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql" event={"ID":"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7","Type":"ContainerDied","Data":"2715b052a868713ab837df340c0bcccf58dfefb6898c037a4af0fd2d80bac898"}
Oct 30 03:53:55.190978 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.190944    4494 generic.go:356] "Generic (PLEG): container finished" podID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerID="0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1" exitCode=0
Oct 30 03:53:55.191110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.191041    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" event={"ID":"83c73456-0715-4de7-b3c1-8a4a991dd6eb","Type":"ContainerDied","Data":"0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1"}
Oct 30 03:53:55.191110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.191063    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"
Oct 30 03:53:55.191110 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.191090    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq" event={"ID":"83c73456-0715-4de7-b3c1-8a4a991dd6eb","Type":"ContainerDied","Data":"d893ddd9a2b9d7130134308a3aa0ef0e977966da67a2659597d366fe87dc9df8"}
Oct 30 03:53:55.192988 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.192965    4494 generic.go:356] "Generic (PLEG): container finished" podID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerID="c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389" exitCode=0
Oct 30 03:53:55.193102 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.193005    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" event={"ID":"ed43124b-6611-49bd-ba1b-48e73fa3c14b","Type":"ContainerDied","Data":"c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389"}
Oct 30 03:53:55.193102 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.193034    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz" event={"ID":"ed43124b-6611-49bd-ba1b-48e73fa3c14b","Type":"ContainerDied","Data":"4d50ff8924de28ca2c298add2674a0ac858ea4b75b5303fa511df57625bfff36"}
Oct 30 03:53:55.193102 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.193093    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"
Oct 30 03:53:55.194039 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.194017    4494 scope.go:122] "RemoveContainer" containerID="a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68"
Oct 30 03:53:55.195413 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:53:55.195392    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68\": container with ID starting with a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68 not found: ID does not exist" containerID="a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68"
Oct 30 03:53:55.195500 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.195419    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68"} err="failed to get container status \"a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68\": rpc error: code = NotFound desc = could not find container \"a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68\": container with ID starting with a76ac4cb9bc341929ec9431c42b799aeb0a7c7f01eb78d6189511ba7a8776a68 not found: ID does not exist"
Oct 30 03:53:55.195500 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.195437    4494 scope.go:122] "RemoveContainer" containerID="46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99"
Oct 30 03:53:55.204787 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.204619    4494 scope.go:122] "RemoveContainer" containerID="46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99"
Oct 30 03:53:55.205034 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:53:55.205013    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99\": container with ID starting with 46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99 not found: ID does not exist" containerID="46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99"
Oct 30 03:53:55.205121 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.205047    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99"} err="failed to get container status \"46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99\": rpc error: code = NotFound desc = could not find container \"46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99\": container with ID starting with 46a780c23ba992a1d74efae927908882b514fa236c63548b6e91b30d2e884f99 not found: ID does not exist"
Oct 30 03:53:55.205121 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.205070    4494 scope.go:122] "RemoveContainer" containerID="5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7"
Oct 30 03:53:55.213306 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.213290    4494 scope.go:122] "RemoveContainer" containerID="5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7"
Oct 30 03:53:55.213551 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:53:55.213528    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7\": container with ID starting with 5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7 not found: ID does not exist" containerID="5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7"
Oct 30 03:53:55.213636 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.213561    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7"} err="failed to get container status \"5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7\": rpc error: code = NotFound desc = could not find container \"5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7\": container with ID starting with 5f83bc73043979ad50f348590e8ac6cee8043d4c22548bd6726b2a21aa43efb7 not found: ID does not exist"
Oct 30 03:53:55.213636 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.213585    4494 scope.go:122] "RemoveContainer" containerID="0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1"
Oct 30 03:53:55.219782 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.219760    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tmp-dir\") pod \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") "
Oct 30 03:53:55.219878 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.219795    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-podinfo\") pod \"51422a6f-9df9-4283-815e-a0dd6660cb3f\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") "
Oct 30 03:53:55.219878 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.219817    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-credential-socket\") pod \"51422a6f-9df9-4283-815e-a0dd6660cb3f\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") "
Oct 30 03:53:55.219878 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.219843    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-home\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-home\") pod \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") "
Oct 30 03:53:55.219878 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.219874    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-envoy\") pod \"51422a6f-9df9-4283-815e-a0dd6660cb3f\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") "
Oct 30 03:53:55.220112 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.219939    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-data\") pod \"51422a6f-9df9-4283-815e-a0dd6660cb3f\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") "
Oct 30 03:53:55.220112 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.219970    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-podinfo\") pod \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") "
Oct 30 03:53:55.220112 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220009    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tls-certs\") pod \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") "
Oct 30 03:53:55.220112 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220035    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-certs\") pod \"51422a6f-9df9-4283-815e-a0dd6660cb3f\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") "
Oct 30 03:53:55.220112 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220071    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-certs\") pod \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") "
Oct 30 03:53:55.220112 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220101    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c73456-0715-4de7-b3c1-8a4a991dd6eb-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83c73456-0715-4de7-b3c1-8a4a991dd6eb-cabundle-cert\") pod \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") "
Oct 30 03:53:55.220416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220133    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51422a6f-9df9-4283-815e-a0dd6660cb3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51422a6f-9df9-4283-815e-a0dd6660cb3f-istiod-ca-cert\") pod \"51422a6f-9df9-4283-815e-a0dd6660cb3f\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") "
Oct 30 03:53:55.220416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220161    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-token\" (UniqueName: \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-token\") pod \"51422a6f-9df9-4283-815e-a0dd6660cb3f\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") "
Oct 30 03:53:55.220416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220181    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kube-api-access-7fhf8\" (UniqueName: \"kubernetes.io/projected/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kube-api-access-7fhf8\") pod \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") "
Oct 30 03:53:55.220416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220223    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-data" pod "51422a6f-9df9-4283-815e-a0dd6660cb3f" (UID: "51422a6f-9df9-4283-815e-a0dd6660cb3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.220416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220230    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-envoy\") pod \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") "
Oct 30 03:53:55.220416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220293    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-socket\") pod \"51422a6f-9df9-4283-815e-a0dd6660cb3f\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") "
Oct 30 03:53:55.220416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220344    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-token\" (UniqueName: \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-token\") pod \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") "
Oct 30 03:53:55.220416 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220398    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-credential-socket\") pod \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") "
Oct 30 03:53:55.221006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220430    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istiod-ca-cert\") pod \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") "
Oct 30 03:53:55.221006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220464    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-data\") pod \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") "
Oct 30 03:53:55.221006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220502    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-kube-api-access-rc28w\" (UniqueName: \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-kube-api-access-rc28w\") pod \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") "
Oct 30 03:53:55.221006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220534    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-dshm\") pod \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") "
Oct 30 03:53:55.221006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220568    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-socket\") pod \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\" (UID: \"7f1bf035-bfcd-4d55-b1b5-b2054d8930e7\") "
Oct 30 03:53:55.221006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220573    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c73456-0715-4de7-b3c1-8a4a991dd6eb-cabundle-cert" pod "83c73456-0715-4de7-b3c1-8a4a991dd6eb" (UID: "83c73456-0715-4de7-b3c1-8a4a991dd6eb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:55.221006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220603    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-kube-api-access-znf72\" (UniqueName: \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-kube-api-access-znf72\") pod \"51422a6f-9df9-4283-815e-a0dd6660cb3f\" (UID: \"51422a6f-9df9-4283-815e-a0dd6660cb3f\") "
Oct 30 03:53:55.221006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220645    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-model-cache\") pod \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") "
Oct 30 03:53:55.221006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220674    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kserve-provision-location\") pod \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\" (UID: \"83c73456-0715-4de7-b3c1-8a4a991dd6eb\") "
Oct 30 03:53:55.221006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220930    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-home" pod "83c73456-0715-4de7-b3c1-8a4a991dd6eb" (UID: "83c73456-0715-4de7-b3c1-8a4a991dd6eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.221505 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221072    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-certs" pod "51422a6f-9df9-4283-815e-a0dd6660cb3f" (UID: "51422a6f-9df9-4283-815e-a0dd6660cb3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.221505 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221405    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-certs" pod "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" (UID: "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.221614 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221513    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51422a6f-9df9-4283-815e-a0dd6660cb3f-istiod-ca-cert" pod "51422a6f-9df9-4283-815e-a0dd6660cb3f" (UID: "51422a6f-9df9-4283-815e-a0dd6660cb3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:55.221614 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221091    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83c73456-0715-4de7-b3c1-8a4a991dd6eb-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221614 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221566    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221614 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221583    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97fvg\" (UniqueName: \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-kube-api-access-97fvg\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221614 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221598    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eca957c-c200-4e82-aab8-ad0806610ed3-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221614 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221612    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221624    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221636    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221650    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221663    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221674    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221685    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221697    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221708    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed43124b-6611-49bd-ba1b-48e73fa3c14b-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221723    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221737    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eca957c-c200-4e82-aab8-ad0806610ed3-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221752    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkn9k\" (UniqueName: \"kubernetes.io/projected/2eca957c-c200-4e82-aab8-ad0806610ed3-kube-api-access-gkn9k\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.221975 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.221766    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed43124b-6611-49bd-ba1b-48e73fa3c14b-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.222624 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.222042    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-data" pod "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" (UID: "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.222624 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.222250    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-socket" pod "51422a6f-9df9-4283-815e-a0dd6660cb3f" (UID: "51422a6f-9df9-4283-815e-a0dd6660cb3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.222624 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.222381    4494 scope.go:122] "RemoveContainer" containerID="d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216"
Oct 30 03:53:55.222624 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.220147    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-credential-socket" pod "51422a6f-9df9-4283-815e-a0dd6660cb3f" (UID: "51422a6f-9df9-4283-815e-a0dd6660cb3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.222830 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.222792    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-credential-socket" pod "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" (UID: "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.223294 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.223032    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-socket" pod "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" (UID: "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.223294 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.223109    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istiod-ca-cert" pod "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" (UID: "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:53:55.224025 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.223832    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-envoy" pod "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" (UID: "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.224452 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.224397    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-model-cache" pod "83c73456-0715-4de7-b3c1-8a4a991dd6eb" (UID: "83c73456-0715-4de7-b3c1-8a4a991dd6eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.225169 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.225134    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-envoy" pod "51422a6f-9df9-4283-815e-a0dd6660cb3f" (UID: "51422a6f-9df9-4283-815e-a0dd6660cb3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.225600 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.225575    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tls-certs" pod "83c73456-0715-4de7-b3c1-8a4a991dd6eb" (UID: "83c73456-0715-4de7-b3c1-8a4a991dd6eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:53:55.227255 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.227232    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kube-api-access-7fhf8" pod "83c73456-0715-4de7-b3c1-8a4a991dd6eb" (UID: "83c73456-0715-4de7-b3c1-8a4a991dd6eb"). InnerVolumeSpecName "kube-api-access-7fhf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.227375 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.227262    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-token" pod "51422a6f-9df9-4283-815e-a0dd6660cb3f" (UID: "51422a6f-9df9-4283-815e-a0dd6660cb3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.227472 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.227315    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-token" pod "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" (UID: "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.227472 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.227362    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-kube-api-access-rc28w" pod "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" (UID: "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7"). InnerVolumeSpecName "kube-api-access-rc28w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.227586 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.227480    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-dshm" pod "83c73456-0715-4de7-b3c1-8a4a991dd6eb" (UID: "83c73456-0715-4de7-b3c1-8a4a991dd6eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.228652 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.228619    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-podinfo" pod "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" (UID: "7f1bf035-bfcd-4d55-b1b5-b2054d8930e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:53:55.228796 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.228780    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-podinfo" pod "51422a6f-9df9-4283-815e-a0dd6660cb3f" (UID: "51422a6f-9df9-4283-815e-a0dd6660cb3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:53:55.228965 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.228944    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-kube-api-access-znf72" pod "51422a6f-9df9-4283-815e-a0dd6660cb3f" (UID: "51422a6f-9df9-4283-815e-a0dd6660cb3f"). InnerVolumeSpecName "kube-api-access-znf72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:53:55.235067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.235038    4494 scope.go:122] "RemoveContainer" containerID="0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1"
Oct 30 03:53:55.235460 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:53:55.235441    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1\": container with ID starting with 0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1 not found: ID does not exist" containerID="0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1"
Oct 30 03:53:55.235545 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.235472    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1"} err="failed to get container status \"0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1\": rpc error: code = NotFound desc = could not find container \"0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1\": container with ID starting with 0c4a084889e90855504b339a633e8391a6b358f2db89ef1e474f191486380ea1 not found: ID does not exist"
Oct 30 03:53:55.235545 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.235496    4494 scope.go:122] "RemoveContainer" containerID="d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216"
Oct 30 03:53:55.235867 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:53:55.235844    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216\": container with ID starting with d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216 not found: ID does not exist" containerID="d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216"
Oct 30 03:53:55.235867 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.235877    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216"} err="failed to get container status \"d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216\": rpc error: code = NotFound desc = could not find container \"d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216\": container with ID starting with d861b48242748cca65d1227b1ee1d44640b970a1ba87fa3e699d15e183584216 not found: ID does not exist"
Oct 30 03:53:55.235867 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.235930    4494 scope.go:122] "RemoveContainer" containerID="c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389"
Oct 30 03:53:55.243093 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.243053    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tmp-dir" pod "83c73456-0715-4de7-b3c1-8a4a991dd6eb" (UID: "83c73456-0715-4de7-b3c1-8a4a991dd6eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.244420 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.244394    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kserve-provision-location" pod "83c73456-0715-4de7-b3c1-8a4a991dd6eb" (UID: "83c73456-0715-4de7-b3c1-8a4a991dd6eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:53:55.245943 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.245924    4494 scope.go:122] "RemoveContainer" containerID="c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389"
Oct 30 03:53:55.246286 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:53:55.246262    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389\": container with ID starting with c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389 not found: ID does not exist" containerID="c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389"
Oct 30 03:53:55.246387 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.246294    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389"} err="failed to get container status \"c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389\": rpc error: code = NotFound desc = could not find container \"c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389\": container with ID starting with c7a60840fbb38fefd31321fbf0d38d75c584f44f0af3b997ec95c7da92925389 not found: ID does not exist"
Oct 30 03:53:55.284397 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.284322    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"]
Oct 30 03:53:55.288228 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.288201    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-7fmnz"]
Oct 30 03:53:55.322997 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.322972    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.322997 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.322996    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323007    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323016    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323025    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323033    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc28w\" (UniqueName: \"kubernetes.io/projected/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-kube-api-access-rc28w\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323042    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323052    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323060    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znf72\" (UniqueName: \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-kube-api-access-znf72\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323068    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323076    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323085    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323092    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323099    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323107    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323115    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323122    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c73456-0715-4de7-b3c1-8a4a991dd6eb-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323127 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323129    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51422a6f-9df9-4283-815e-a0dd6660cb3f-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323626 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323137    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323626 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323144    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51422a6f-9df9-4283-815e-a0dd6660cb3f-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323626 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323151    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51422a6f-9df9-4283-815e-a0dd6660cb3f-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323626 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323161    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fhf8\" (UniqueName: \"kubernetes.io/projected/83c73456-0715-4de7-b3c1-8a4a991dd6eb-kube-api-access-7fhf8\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.323626 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.323169    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:53:55.337089 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.337063    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"]
Oct 30 03:53:55.341072 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.341047    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-85p64"]
Oct 30 03:53:55.532695 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.532663    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"]
Oct 30 03:53:55.536263 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.536199    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-bbkql"]
Oct 30 03:53:55.570691 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.570653    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"]
Oct 30 03:53:55.576429 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.576396    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n5z65"]
Oct 30 03:53:55.611120 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.611085    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"]
Oct 30 03:53:55.615688 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.615661    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-9f98557f7-k8kqq"]
Oct 30 03:53:55.926717 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.926630    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" path="/var/lib/kubelet/pods/2eca957c-c200-4e82-aab8-ad0806610ed3/volumes"
Oct 30 03:53:55.927188 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.927167    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" path="/var/lib/kubelet/pods/51422a6f-9df9-4283-815e-a0dd6660cb3f/volumes"
Oct 30 03:53:55.927633 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.927607    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" path="/var/lib/kubelet/pods/7f1bf035-bfcd-4d55-b1b5-b2054d8930e7/volumes"
Oct 30 03:53:55.928448 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.928430    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" path="/var/lib/kubelet/pods/83c73456-0715-4de7-b3c1-8a4a991dd6eb/volumes"
Oct 30 03:53:55.928841 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:53:55.928829    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" path="/var/lib/kubelet/pods/ed43124b-6611-49bd-ba1b-48e73fa3c14b/volumes"
Oct 30 03:54:24.082599 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:54:24.082555    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:54:28.901713 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.901676    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"]
Oct 30 03:54:28.902138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902063    4494 cpu_manager.go:414] "containerMap: removing container" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerName="istio-proxy"
Oct 30 03:54:28.902138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902075    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerName="istio-proxy"
Oct 30 03:54:28.902138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902086    4494 cpu_manager.go:414] "containerMap: removing container" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerName="istio-proxy"
Oct 30 03:54:28.902138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902091    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerName="istio-proxy"
Oct 30 03:54:28.902138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902098    4494 cpu_manager.go:414] "containerMap: removing container" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="storage-initializer"
Oct 30 03:54:28.902138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902102    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="storage-initializer"
Oct 30 03:54:28.902138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902108    4494 cpu_manager.go:414] "containerMap: removing container" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="storage-initializer"
Oct 30 03:54:28.902138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902112    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="storage-initializer"
Oct 30 03:54:28.902138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902122    4494 cpu_manager.go:414] "containerMap: removing container" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerName="istio-proxy"
Oct 30 03:54:28.902138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902127    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerName="istio-proxy"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902146    4494 cpu_manager.go:414] "containerMap: removing container" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902151    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902161    4494 cpu_manager.go:414] "containerMap: removing container" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerName="istio-proxy"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902166    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerName="istio-proxy"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902172    4494 cpu_manager.go:414] "containerMap: removing container" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902176    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902241    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe96f39-f9b7-4c50-8d42-e041fb998571" containerName="main"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902251    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="51422a6f-9df9-4283-815e-a0dd6660cb3f" containerName="istio-proxy"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902258    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c73456-0715-4de7-b3c1-8a4a991dd6eb" containerName="main"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902264    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed43124b-6611-49bd-ba1b-48e73fa3c14b" containerName="istio-proxy"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902270    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1bf035-bfcd-4d55-b1b5-b2054d8930e7" containerName="istio-proxy"
Oct 30 03:54:28.902441 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.902275    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eca957c-c200-4e82-aab8-ad0806610ed3" containerName="istio-proxy"
Oct 30 03:54:28.910346 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.910310    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:28.930461 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.930415    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Oct 30 03:54:28.932793 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.932757    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"]
Oct 30 03:54:28.942635 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.942604    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Oct 30 03:54:28.942635 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.942604    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Oct 30 03:54:28.942853 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:28.942711    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-qrtgt\""
Oct 30 03:54:29.011129 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.011084    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-certs\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.011329 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.011153    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qthnp\" (UniqueName: \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-kube-api-access-qthnp\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.011329 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.011216    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.011329 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.011242    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-podinfo\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.011329 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.011259    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-envoy\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.011329 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.011278    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-data\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.011329 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.011317    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.011536 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.011413    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-credential-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.011536 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.011432    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-token\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.112881 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.112842    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-credential-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.112881 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.112879    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-token\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113160 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.112937    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-certs\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113160 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.112967    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qthnp\" (UniqueName: \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-kube-api-access-qthnp\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113160 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.112984    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113160 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.113004    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-podinfo\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113160 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.113020    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-envoy\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113160 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.113037    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-data\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113160 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.113055    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113470 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.113359    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-certs\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113470 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.113386    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-credential-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113741 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.113714    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-socket\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113842 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.113751    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-data\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.113842 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.113789    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.115624 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.115601    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-envoy\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.115737 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.115720    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-podinfo\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.137722 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.137675    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-token\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.138810 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.138784    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qthnp\" (UniqueName: \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-kube-api-access-qthnp\") pod \"router-gateway-1-openshift-default-56d96d6b69-vpwtl\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.225937 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.225802    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:29.541562 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.541531    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"]
Oct 30 03:54:29.544127 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:54:29.544092    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6f815f_53db_4f8e_bb34_0feff9ebd5ce.slice/crio-2bf96e783fff16d5d0fe8f1279f30ee54f027e351aa74b64acd9b8a31da8d45a WatchSource:0}: Error finding container 2bf96e783fff16d5d0fe8f1279f30ee54f027e351aa74b64acd9b8a31da8d45a: Status 404 returned error can't find the container with id 2bf96e783fff16d5d0fe8f1279f30ee54f027e351aa74b64acd9b8a31da8d45a
Oct 30 03:54:29.546227 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.546197    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:54:29.546302 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.546270    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:54:29.546343 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:29.546301    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:54:30.358155 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:30.358109    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" event={"ID":"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce","Type":"ContainerStarted","Data":"e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7"}
Oct 30 03:54:30.358573 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:30.358185    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" event={"ID":"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce","Type":"ContainerStarted","Data":"2bf96e783fff16d5d0fe8f1279f30ee54f027e351aa74b64acd9b8a31da8d45a"}
Oct 30 03:54:30.417425 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:30.417363    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" podStartSLOduration=2.417347444 podStartE2EDuration="2.417347444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:30.416183839 +0000 UTC m=+3534.622364065" watchObservedRunningTime="2026-10-30 03:54:30.417347444 +0000 UTC m=+3534.623527662"
Oct 30 03:54:31.202356 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.202312    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"]
Oct 30 03:54:31.206671 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.206650    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.226454 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.226416    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:31.227453 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.227427    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:54:31.231933 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.231874    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:31.239949 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.239879    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Oct 30 03:54:31.240084 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.239969    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Oct 30 03:54:31.243001 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.242968    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-lhj4b\""
Oct 30 03:54:31.244503 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.244480    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"]
Oct 30 03:54:31.335327 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.335281    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-cabundle-cert\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.335529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.335333    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-home\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.335529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.335413    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tls-certs\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.335529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.335452    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tmp-dir\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.335529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.335507    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kserve-provision-location\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.335739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.335560    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sghlx\" (UniqueName: \"kubernetes.io/projected/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kube-api-access-sghlx\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.335739 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.335604    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-model-cache\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.335833 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.335733    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-dshm\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.364048 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.364010    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:31.365033 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.365011    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:31.436814 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.436772    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sghlx\" (UniqueName: \"kubernetes.io/projected/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kube-api-access-sghlx\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437033 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.436834    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-model-cache\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437033 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.436876    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-dshm\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437164 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.437052    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-cabundle-cert\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437164 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.437149    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-home\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437281 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.437180    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tls-certs\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437281 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.437215    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tmp-dir\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437281 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.437247    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kserve-provision-location\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437454 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.437309    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-model-cache\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437535 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.437507    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-home\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437619 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.437594    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kserve-provision-location\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437746 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.437689    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tmp-dir\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.437846 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.437825    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-cabundle-cert\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.440255 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.440225    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-dshm\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.440357 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.440316    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tls-certs\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.461330 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.461254    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sghlx\" (UniqueName: \"kubernetes.io/projected/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kube-api-access-sghlx\") pod \"router-with-refs-test-kserve-c4b85d989-sfwrx\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.523118 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.523072    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:31.839506 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:31.839479    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"]
Oct 30 03:54:31.842114 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:54:31.842085    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c1bfe0_9356_4043_bae1_03cd1f3882ca.slice/crio-c1beffed25bc2e5502f609dc66aa64c610fcd58a79ac7b0ce5676e56c6f16d90 WatchSource:0}: Error finding container c1beffed25bc2e5502f609dc66aa64c610fcd58a79ac7b0ce5676e56c6f16d90: Status 404 returned error can't find the container with id c1beffed25bc2e5502f609dc66aa64c610fcd58a79ac7b0ce5676e56c6f16d90
Oct 30 03:54:32.369987 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:32.369935    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" event={"ID":"b2c1bfe0-9356-4043-bae1-03cd1f3882ca","Type":"ContainerStarted","Data":"035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693"}
Oct 30 03:54:32.370505 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:32.370012    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" event={"ID":"b2c1bfe0-9356-4043-bae1-03cd1f3882ca","Type":"ContainerStarted","Data":"c1beffed25bc2e5502f609dc66aa64c610fcd58a79ac7b0ce5676e56c6f16d90"}
Oct 30 03:54:35.406271 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:35.406225    4494 generic.go:356] "Generic (PLEG): container finished" podID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerID="035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693" exitCode=0
Oct 30 03:54:35.406780 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:35.406291    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" event={"ID":"b2c1bfe0-9356-4043-bae1-03cd1f3882ca","Type":"ContainerDied","Data":"035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693"}
Oct 30 03:54:36.414540 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:36.414501    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" event={"ID":"b2c1bfe0-9356-4043-bae1-03cd1f3882ca","Type":"ContainerStarted","Data":"25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73"}
Oct 30 03:54:36.488938 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:36.488452    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podStartSLOduration=2.812176209 podStartE2EDuration="5.488429109s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.6762529s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:36.473150847 +0000 UTC m=+3540.679331070" watchObservedRunningTime="2026-10-30 03:54:36.488429109 +0000 UTC m=+3540.694609330"
Oct 30 03:54:36.523757 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:36.523713    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:41.442986 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:41.442939    4494 generic.go:356] "Generic (PLEG): container finished" podID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerID="e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7" exitCode=0
Oct 30 03:54:41.443444 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:41.443013    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" event={"ID":"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce","Type":"ContainerDied","Data":"e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7"}
Oct 30 03:54:41.443444 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:41.443396    4494 scope.go:122] "RemoveContainer" containerID="e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7"
Oct 30 03:54:41.444681 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:41.444645    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:54:41.444782 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:41.444722    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:54:41.444782 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:41.444756    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 03:54:41.523465 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:41.523432    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:54:41.525370 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:41.525333    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:54:42.450283 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:42.450248    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" event={"ID":"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce","Type":"ContainerStarted","Data":"e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49"}
Oct 30 03:54:43.226405 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:43.226362    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:43.227584 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:43.227548    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 03:54:44.226689 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:44.226636    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 03:54:45.226751 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:45.226696    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 03:54:46.226879 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:46.226828    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 03:54:46.364194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:46.364153    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:47.227276 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:47.227222    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 03:54:48.230875 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:48.230838    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:48.231996 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:48.231974    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:54:51.524174 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:54:51.524126    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:54:54.083610 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:54:54.083568    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:55:01.523990 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:01.523941    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:55:11.524368 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:11.524310    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:55:21.523829 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:21.523774    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:55:24.082809 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:55:24.082771    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:55:31.524648 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:31.524592    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:55:41.523976 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:41.523917    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:55:46.398382 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.398343    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"]
Oct 30 03:55:46.402106 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.402079    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.421018 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.420988    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"]
Oct 30 03:55:46.430461 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.430419    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Oct 30 03:55:46.430461 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.430436    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:55:46.430461 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.430423    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Oct 30 03:55:46.430461 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.430422    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Oct 30 03:55:46.430747 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.430509    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-n4xll\""
Oct 30 03:55:46.430747 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.430618    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Oct 30 03:55:46.449655 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.449618    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-home\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.449655 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.449657    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tmp-dir\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.449849 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.449677    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tls-certs\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.449849 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.449739    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-model-cache\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.449849 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.449830    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cee40dd7-e512-4a34-8fa3-2680d072a4f0-cabundle-cert\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.449984 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.449918    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-dshm\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.449984 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.449966    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kserve-provision-location\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.449984 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.449983    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw2g9\" (UniqueName: \"kubernetes.io/projected/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kube-api-access-rw2g9\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.550817 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.550768    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cee40dd7-e512-4a34-8fa3-2680d072a4f0-cabundle-cert\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.550835    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-dshm\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.550922    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kserve-provision-location\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.550946    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw2g9\" (UniqueName: \"kubernetes.io/projected/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kube-api-access-rw2g9\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.550979    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-home\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551066 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.551006    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tmp-dir\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551334 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.551069    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tls-certs\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551334 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.551109    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-model-cache\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551438 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.551358    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-home\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551494 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.551437    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tmp-dir\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551494 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.551477    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kserve-provision-location\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551631 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.551608    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cee40dd7-e512-4a34-8fa3-2680d072a4f0-cabundle-cert\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.551803 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.551780    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-model-cache\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.553758 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.553730    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-dshm\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.553944 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.553927    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tls-certs\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.576879 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.576838    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw2g9\" (UniqueName: \"kubernetes.io/projected/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kube-api-access-rw2g9\") pod \"stop-feature-test-kserve-85bf77f655-fg6hp\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:46.719803 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:46.719706    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:47.042692 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:47.042659    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"]
Oct 30 03:55:47.047504 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:55:47.047391    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee40dd7_e512_4a34_8fa3_2680d072a4f0.slice/crio-75313c7df3d568f67f24121e4fe2609e49635bd48771b169fffd793fca94c2fa WatchSource:0}: Error finding container 75313c7df3d568f67f24121e4fe2609e49635bd48771b169fffd793fca94c2fa: Status 404 returned error can't find the container with id 75313c7df3d568f67f24121e4fe2609e49635bd48771b169fffd793fca94c2fa
Oct 30 03:55:47.760807 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:47.760703    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" event={"ID":"cee40dd7-e512-4a34-8fa3-2680d072a4f0","Type":"ContainerStarted","Data":"715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a"}
Oct 30 03:55:47.760807 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:47.760763    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" event={"ID":"cee40dd7-e512-4a34-8fa3-2680d072a4f0","Type":"ContainerStarted","Data":"75313c7df3d568f67f24121e4fe2609e49635bd48771b169fffd793fca94c2fa"}
Oct 30 03:55:49.772451 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:49.772409    4494 generic.go:356] "Generic (PLEG): container finished" podID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerID="715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a" exitCode=0
Oct 30 03:55:49.772956 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:49.772483    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" event={"ID":"cee40dd7-e512-4a34-8fa3-2680d072a4f0","Type":"ContainerDied","Data":"715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a"}
Oct 30 03:55:50.780227 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:50.780178    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" event={"ID":"cee40dd7-e512-4a34-8fa3-2680d072a4f0","Type":"ContainerStarted","Data":"6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19"}
Oct 30 03:55:50.852654 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:50.852599    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podStartSLOduration=2.528606568 podStartE2EDuration="4.852583255s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.323976687s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:50.839279089 +0000 UTC m=+3615.045459341" watchObservedRunningTime="2026-10-30 03:55:50.852583255 +0000 UTC m=+3615.058763472"
Oct 30 03:55:51.524204 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:51.524161    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:55:51.720859 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:51.720809    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:54.083797 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:55:54.083760    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:55:56.720125 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:56.720082    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:55:56.721855 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:55:56.721817    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:56:01.524111 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:01.524070    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:56:06.720600 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:06.720556    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:56:11.524511 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:11.524462    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:56:16.720825 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:16.720765    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:56:21.524107 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:21.524005    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Oct 30 03:56:24.085178 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:56:24.085135    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:56:26.720642 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:26.720589    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:56:31.539766 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:31.539734    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:56:31.547822 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:31.547787    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:56:36.720688 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:36.720637    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:56:37.619082 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:37.619050    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"]
Oct 30 03:56:46.720833 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:46.720778    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:56:52.642924 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:52.642856    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main" containerID="cri-o://25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73" gracePeriod=45
Oct 30 03:56:53.390199 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.390171    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:56:53.482496 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.482403    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kube-api-access-sghlx\" (UniqueName: \"kubernetes.io/projected/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kube-api-access-sghlx\") pod \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") "
Oct 30 03:56:53.482673 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.482499    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kserve-provision-location\") pod \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") "
Oct 30 03:56:53.482673 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.482549    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-cabundle-cert\") pod \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") "
Oct 30 03:56:53.482673 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.482609    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-dshm\") pod \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") "
Oct 30 03:56:53.482673 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.482640    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tls-certs\") pod \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") "
Oct 30 03:56:53.482673 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.482668    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tmp-dir\") pod \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") "
Oct 30 03:56:53.483009 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.482710    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-model-cache\") pod \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") "
Oct 30 03:56:53.483009 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.482784    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-home\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-home\") pod \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\" (UID: \"b2c1bfe0-9356-4043-bae1-03cd1f3882ca\") "
Oct 30 03:56:53.483508 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.483309    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-cabundle-cert" pod "b2c1bfe0-9356-4043-bae1-03cd1f3882ca" (UID: "b2c1bfe0-9356-4043-bae1-03cd1f3882ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:53.483633 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.483539    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-model-cache" pod "b2c1bfe0-9356-4043-bae1-03cd1f3882ca" (UID: "b2c1bfe0-9356-4043-bae1-03cd1f3882ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:53.483992 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.483964    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-home" pod "b2c1bfe0-9356-4043-bae1-03cd1f3882ca" (UID: "b2c1bfe0-9356-4043-bae1-03cd1f3882ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:53.485502 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.485469    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kube-api-access-sghlx" pod "b2c1bfe0-9356-4043-bae1-03cd1f3882ca" (UID: "b2c1bfe0-9356-4043-bae1-03cd1f3882ca"). InnerVolumeSpecName "kube-api-access-sghlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:53.485637 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.485619    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tls-certs" pod "b2c1bfe0-9356-4043-bae1-03cd1f3882ca" (UID: "b2c1bfe0-9356-4043-bae1-03cd1f3882ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:53.485637 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.485621    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-dshm" pod "b2c1bfe0-9356-4043-bae1-03cd1f3882ca" (UID: "b2c1bfe0-9356-4043-bae1-03cd1f3882ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:53.499720 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.499695    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tmp-dir" pod "b2c1bfe0-9356-4043-bae1-03cd1f3882ca" (UID: "b2c1bfe0-9356-4043-bae1-03cd1f3882ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:53.515608 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.515568    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kserve-provision-location" pod "b2c1bfe0-9356-4043-bae1-03cd1f3882ca" (UID: "b2c1bfe0-9356-4043-bae1-03cd1f3882ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:53.583995 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.583962    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:56:53.583995 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.583995    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sghlx\" (UniqueName: \"kubernetes.io/projected/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kube-api-access-sghlx\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:56:53.584198 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.584006    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:56:53.584198 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.584016    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:56:53.584198 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.584024    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:56:53.584198 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.584032    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:56:53.584198 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.584039    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:56:53.584198 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:53.584046    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c1bfe0-9356-4043-bae1-03cd1f3882ca-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:56:54.082365 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:56:54.082324    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:56:54.094504 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.094469    4494 generic.go:356] "Generic (PLEG): container finished" podID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerID="25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73" exitCode=0
Oct 30 03:56:54.094654 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.094527    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"
Oct 30 03:56:54.094654 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.094545    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" event={"ID":"b2c1bfe0-9356-4043-bae1-03cd1f3882ca","Type":"ContainerDied","Data":"25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73"}
Oct 30 03:56:54.094654 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.094605    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx" event={"ID":"b2c1bfe0-9356-4043-bae1-03cd1f3882ca","Type":"ContainerDied","Data":"c1beffed25bc2e5502f609dc66aa64c610fcd58a79ac7b0ce5676e56c6f16d90"}
Oct 30 03:56:54.094654 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.094628    4494 scope.go:122] "RemoveContainer" containerID="25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73"
Oct 30 03:56:54.103086 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.103067    4494 scope.go:122] "RemoveContainer" containerID="035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693"
Oct 30 03:56:54.111078 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.111049    4494 scope.go:122] "RemoveContainer" containerID="25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73"
Oct 30 03:56:54.111298 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:56:54.111274    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73\": container with ID starting with 25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73 not found: ID does not exist" containerID="25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73"
Oct 30 03:56:54.111378 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.111311    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73"} err="failed to get container status \"25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73\": rpc error: code = NotFound desc = could not find container \"25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73\": container with ID starting with 25ac188e47d140e799fa80be89623585d88e0a94da423aad31faa9a64910cc73 not found: ID does not exist"
Oct 30 03:56:54.111378 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.111332    4494 scope.go:122] "RemoveContainer" containerID="035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693"
Oct 30 03:56:54.111614 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:56:54.111595    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693\": container with ID starting with 035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693 not found: ID does not exist" containerID="035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693"
Oct 30 03:56:54.111703 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.111623    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693"} err="failed to get container status \"035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693\": rpc error: code = NotFound desc = could not find container \"035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693\": container with ID starting with 035a5498d9feab1c02415edf69488bdb9e626568a1399d315466d93647bd4693 not found: ID does not exist"
Oct 30 03:56:54.158965 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.158924    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"]
Oct 30 03:56:54.171685 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:54.171654    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-c4b85d989-sfwrx"]
Oct 30 03:56:55.927204 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:55.927163    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" path="/var/lib/kubelet/pods/b2c1bfe0-9356-4043-bae1-03cd1f3882ca/volumes"
Oct 30 03:56:56.720640 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:56:56.720602    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:57:06.720614 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:06.720567    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:57:16.720842 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:16.720798    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:57:24.081021 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:57:24.080976    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:57:26.721106 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:26.721065    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:57:31.728297 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.728258    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"]
Oct 30 03:57:31.728797 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.728780    4494 cpu_manager.go:414] "containerMap: removing container" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="storage-initializer"
Oct 30 03:57:31.728844 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.728800    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="storage-initializer"
Oct 30 03:57:31.728844 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.728812    4494 cpu_manager.go:414] "containerMap: removing container" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main"
Oct 30 03:57:31.728844 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.728820    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main"
Oct 30 03:57:31.728977 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.728964    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c1bfe0-9356-4043-bae1-03cd1f3882ca" containerName="main"
Oct 30 03:57:31.735320 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.735296    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.746241 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.746169    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"]
Oct 30 03:57:31.756622 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.756590    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Oct 30 03:57:31.756743 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.756699    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-wst2f\""
Oct 30 03:57:31.756815 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.756778    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Oct 30 03:57:31.811321 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.811295    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:57:31.811465 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.811351    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Oct 30 03:57:31.811465 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.811354    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Oct 30 03:57:31.833806 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.833782    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95644d78-79a3-48e4-8035-6905dd99d92d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.833943 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.833812    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.833943 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.833934    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx5cr\" (UniqueName: \"kubernetes.io/projected/95644d78-79a3-48e4-8035-6905dd99d92d-kube-api-access-xx5cr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.834038 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.834020    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.834078 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.834062    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.834138 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.834125    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.834174 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.834147    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.834216 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.834205    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95644d78-79a3-48e4-8035-6905dd99d92d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.935230 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935183    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95644d78-79a3-48e4-8035-6905dd99d92d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.935230 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935228    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95644d78-79a3-48e4-8035-6905dd99d92d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.935521 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935245    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.935521 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935296    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx5cr\" (UniqueName: \"kubernetes.io/projected/95644d78-79a3-48e4-8035-6905dd99d92d-kube-api-access-xx5cr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.935521 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935316    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.935521 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935345    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.935521 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935390    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.935521 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935412    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.935854 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935829    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.935963 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935939    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.936039 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.935985    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.936101 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.936046    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.936101 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.936053    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95644d78-79a3-48e4-8035-6905dd99d92d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.938135 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.938106    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.938135 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.938123    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95644d78-79a3-48e4-8035-6905dd99d92d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:31.962276 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:31.962241    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx5cr\" (UniqueName: \"kubernetes.io/projected/95644d78-79a3-48e4-8035-6905dd99d92d-kube-api-access-xx5cr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:32.056962 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:32.056922    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:32.382877 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:32.382849    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"]
Oct 30 03:57:32.383420 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:57:32.383390    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95644d78_79a3_48e4_8035_6905dd99d92d.slice/crio-80f564167a5b24b164cddcd999a41d38bb6db1bc2e7070a951c9d04650ee1110 WatchSource:0}: Error finding container 80f564167a5b24b164cddcd999a41d38bb6db1bc2e7070a951c9d04650ee1110: Status 404 returned error can't find the container with id 80f564167a5b24b164cddcd999a41d38bb6db1bc2e7070a951c9d04650ee1110
Oct 30 03:57:32.712520 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:32.712429    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"]
Oct 30 03:57:32.712718 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:32.712687    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy" containerID="cri-o://e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49" gracePeriod=30
Oct 30 03:57:33.233728 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:33.233681    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 03:57:33.275803 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:33.275766    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" event={"ID":"95644d78-79a3-48e4-8035-6905dd99d92d","Type":"ContainerStarted","Data":"117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5"}
Oct 30 03:57:33.276006 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:33.275840    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" event={"ID":"95644d78-79a3-48e4-8035-6905dd99d92d","Type":"ContainerStarted","Data":"80f564167a5b24b164cddcd999a41d38bb6db1bc2e7070a951c9d04650ee1110"}
Oct 30 03:57:35.291444 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:35.291409    4494 generic.go:356] "Generic (PLEG): container finished" podID="95644d78-79a3-48e4-8035-6905dd99d92d" containerID="117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5" exitCode=0
Oct 30 03:57:35.291827 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:35.291480    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" event={"ID":"95644d78-79a3-48e4-8035-6905dd99d92d","Type":"ContainerDied","Data":"117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5"}
Oct 30 03:57:36.298893 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:36.298850    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" event={"ID":"95644d78-79a3-48e4-8035-6905dd99d92d","Type":"ContainerStarted","Data":"5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4"}
Oct 30 03:57:36.372563 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:36.372500    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podStartSLOduration=3.139230455 podStartE2EDuration="5.372482191s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.233251736s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:36.359088604 +0000 UTC m=+3720.565268822" watchObservedRunningTime="2026-10-30 03:57:36.372482191 +0000 UTC m=+3720.578662410"
Oct 30 03:57:36.721141 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:36.721101    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 30 03:57:37.970263 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:37.970237    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:57:38.098277 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098236    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-credential-socket\") pod \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") "
Oct 30 03:57:38.098429 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098308    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-certs\") pod \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") "
Oct 30 03:57:38.098429 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098350    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-kube-api-access-qthnp\" (UniqueName: \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-kube-api-access-qthnp\") pod \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") "
Oct 30 03:57:38.098429 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098399    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-envoy\") pod \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") "
Oct 30 03:57:38.098573 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098438    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-socket\") pod \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") "
Oct 30 03:57:38.098573 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098474    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-data\") pod \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") "
Oct 30 03:57:38.098573 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098529    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istiod-ca-cert\") pod \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") "
Oct 30 03:57:38.098669 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098611    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-token\" (UniqueName: \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-token\") pod \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") "
Oct 30 03:57:38.098669 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098612    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-certs" pod "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" (UID: "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:38.098669 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098654    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-podinfo\") pod \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\" (UID: \"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce\") "
Oct 30 03:57:38.098865 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098812    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-data" pod "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" (UID: "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:38.099335 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.098994    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:57:38.099335 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.099013    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:57:38.099335 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.099174    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-socket" pod "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" (UID: "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:38.099335 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.099197    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istiod-ca-cert" pod "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" (UID: "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:57:38.099563 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.099343    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-credential-socket" pod "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" (UID: "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:38.101888 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.101863    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-podinfo" pod "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" (UID: "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 03:57:38.101888 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.101880    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-envoy" pod "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" (UID: "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:57:38.102042 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.101888    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-token" pod "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" (UID: "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:38.102042 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.102017    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-kube-api-access-qthnp" pod "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" (UID: "8a6f815f-53db-4f8e-bb34-0feff9ebd5ce"). InnerVolumeSpecName "kube-api-access-qthnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:57:38.199980 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.199934    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qthnp\" (UniqueName: \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-kube-api-access-qthnp\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:57:38.199980 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.199979    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:57:38.200194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.199998    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:57:38.200194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.200012    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:57:38.200194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.200025    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:57:38.200194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.200037    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:57:38.200194 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.200049    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:57:38.311283 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.311252    4494 generic.go:356] "Generic (PLEG): container finished" podID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerID="e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49" exitCode=0
Oct 30 03:57:38.311477 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.311332    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"
Oct 30 03:57:38.311477 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.311332    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" event={"ID":"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce","Type":"ContainerDied","Data":"e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49"}
Oct 30 03:57:38.311477 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.311462    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl" event={"ID":"8a6f815f-53db-4f8e-bb34-0feff9ebd5ce","Type":"ContainerDied","Data":"2bf96e783fff16d5d0fe8f1279f30ee54f027e351aa74b64acd9b8a31da8d45a"}
Oct 30 03:57:38.311604 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.311489    4494 scope.go:122] "RemoveContainer" containerID="e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49"
Oct 30 03:57:38.322468 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.322448    4494 scope.go:122] "RemoveContainer" containerID="e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7"
Oct 30 03:57:38.332824 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.332799    4494 scope.go:122] "RemoveContainer" containerID="e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49"
Oct 30 03:57:38.333195 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:57:38.333167    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49\": container with ID starting with e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49 not found: ID does not exist" containerID="e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49"
Oct 30 03:57:38.333291 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.333202    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49"} err="failed to get container status \"e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49\": rpc error: code = NotFound desc = could not find container \"e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49\": container with ID starting with e1f744793bb19de08ea1d80856d6d06ba5eb6c3592e82b7020d78f47e49bee49 not found: ID does not exist"
Oct 30 03:57:38.333291 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.333220    4494 scope.go:122] "RemoveContainer" containerID="e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7"
Oct 30 03:57:38.333463 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:57:38.333448    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7\": container with ID starting with e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7 not found: ID does not exist" containerID="e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7"
Oct 30 03:57:38.333516 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.333468    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7"} err="failed to get container status \"e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7\": rpc error: code = NotFound desc = could not find container \"e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7\": container with ID starting with e139b7c7af7a1fab728da27e406b50a8b002e0ed3523a7cbef63134f8c716dc7 not found: ID does not exist"
Oct 30 03:57:38.363610 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.363567    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"]
Oct 30 03:57:38.368803 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:38.368773    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56d96d6b69-vpwtl"]
Oct 30 03:57:39.929254 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:39.929220    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" path="/var/lib/kubelet/pods/8a6f815f-53db-4f8e-bb34-0feff9ebd5ce/volumes"
Oct 30 03:57:42.057967 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:42.057933    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:42.058361 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:42.058007    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:57:42.059529 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:42.059502    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:57:46.730539 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:46.730507    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:57:46.738150 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:46.738118    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:57:49.020260 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:49.020183    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"]
Oct 30 03:57:52.057526 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:57:52.057485    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:57:54.081396 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:57:54.081359    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:58:02.057633 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:02.057582    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:58:04.044940 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.044868    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" containerID="cri-o://6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19" gracePeriod=45
Oct 30 03:58:04.802941 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.802848    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:58:04.840534 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.840499    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tmp-dir\") pod \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") "
Oct 30 03:58:04.840747 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.840543    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kserve-provision-location\") pod \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") "
Oct 30 03:58:04.840747 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.840567    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kube-api-access-rw2g9\" (UniqueName: \"kubernetes.io/projected/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kube-api-access-rw2g9\") pod \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") "
Oct 30 03:58:04.840747 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.840593    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-model-cache\") pod \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") "
Oct 30 03:58:04.840747 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.840619    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee40dd7-e512-4a34-8fa3-2680d072a4f0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cee40dd7-e512-4a34-8fa3-2680d072a4f0-cabundle-cert\") pod \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") "
Oct 30 03:58:04.840747 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.840641    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-dshm\") pod \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") "
Oct 30 03:58:04.841063 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.840863    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-model-cache" pod "cee40dd7-e512-4a34-8fa3-2680d072a4f0" (UID: "cee40dd7-e512-4a34-8fa3-2680d072a4f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:04.841124 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.841046    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee40dd7-e512-4a34-8fa3-2680d072a4f0-cabundle-cert" pod "cee40dd7-e512-4a34-8fa3-2680d072a4f0" (UID: "cee40dd7-e512-4a34-8fa3-2680d072a4f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:04.843152 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.843114    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kube-api-access-rw2g9" pod "cee40dd7-e512-4a34-8fa3-2680d072a4f0" (UID: "cee40dd7-e512-4a34-8fa3-2680d072a4f0"). InnerVolumeSpecName "kube-api-access-rw2g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:04.843618 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.843589    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-dshm" pod "cee40dd7-e512-4a34-8fa3-2680d072a4f0" (UID: "cee40dd7-e512-4a34-8fa3-2680d072a4f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:04.854578 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.854536    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tmp-dir" pod "cee40dd7-e512-4a34-8fa3-2680d072a4f0" (UID: "cee40dd7-e512-4a34-8fa3-2680d072a4f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:04.860713 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.860680    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kserve-provision-location" pod "cee40dd7-e512-4a34-8fa3-2680d072a4f0" (UID: "cee40dd7-e512-4a34-8fa3-2680d072a4f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:04.942099 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.942048    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tls-certs\") pod \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") "
Oct 30 03:58:04.942301 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.942118    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-home\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-home\") pod \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\" (UID: \"cee40dd7-e512-4a34-8fa3-2680d072a4f0\") "
Oct 30 03:58:04.942399 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.942380    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:58:04.942466 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.942403    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cee40dd7-e512-4a34-8fa3-2680d072a4f0-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:58:04.942466 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.942417    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:58:04.942466 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.942429    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:58:04.942633 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.942478    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:58:04.942633 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.942512    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw2g9\" (UniqueName: \"kubernetes.io/projected/cee40dd7-e512-4a34-8fa3-2680d072a4f0-kube-api-access-rw2g9\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:58:04.942744 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.942696    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-home" pod "cee40dd7-e512-4a34-8fa3-2680d072a4f0" (UID: "cee40dd7-e512-4a34-8fa3-2680d072a4f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:04.944348 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:04.944320    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tls-certs" pod "cee40dd7-e512-4a34-8fa3-2680d072a4f0" (UID: "cee40dd7-e512-4a34-8fa3-2680d072a4f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:05.043705 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.043650    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee40dd7-e512-4a34-8fa3-2680d072a4f0-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:58:05.043705 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.043683    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee40dd7-e512-4a34-8fa3-2680d072a4f0-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:58:05.456542 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.456507    4494 generic.go:356] "Generic (PLEG): container finished" podID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerID="6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19" exitCode=0
Oct 30 03:58:05.457008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.456553    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" event={"ID":"cee40dd7-e512-4a34-8fa3-2680d072a4f0","Type":"ContainerDied","Data":"6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19"}
Oct 30 03:58:05.457008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.456627    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" event={"ID":"cee40dd7-e512-4a34-8fa3-2680d072a4f0","Type":"ContainerDied","Data":"75313c7df3d568f67f24121e4fe2609e49635bd48771b169fffd793fca94c2fa"}
Oct 30 03:58:05.457008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.456574    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"
Oct 30 03:58:05.457008 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.456664    4494 scope.go:122] "RemoveContainer" containerID="6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19"
Oct 30 03:58:05.466768 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.466747    4494 scope.go:122] "RemoveContainer" containerID="715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a"
Oct 30 03:58:05.475612 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.475591    4494 scope.go:122] "RemoveContainer" containerID="6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19"
Oct 30 03:58:05.475977 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:58:05.475953    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19\": container with ID starting with 6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19 not found: ID does not exist" containerID="6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19"
Oct 30 03:58:05.476090 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.475984    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19"} err="failed to get container status \"6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19\": rpc error: code = NotFound desc = could not find container \"6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19\": container with ID starting with 6d933b96e7358e8cc9f06e02181b17ecd2089b83fcf68b5cac939f304b8e4d19 not found: ID does not exist"
Oct 30 03:58:05.476090 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.476005    4494 scope.go:122] "RemoveContainer" containerID="715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a"
Oct 30 03:58:05.476316 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:58:05.476296    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a\": container with ID starting with 715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a not found: ID does not exist" containerID="715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a"
Oct 30 03:58:05.476353 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.476327    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a"} err="failed to get container status \"715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a\": rpc error: code = NotFound desc = could not find container \"715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a\": container with ID starting with 715d2c8de1a317ee207776373ef430eb01c403ba4899606cce804f6a46c6de4a not found: ID does not exist"
Oct 30 03:58:05.520709 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.520672    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"]
Oct 30 03:58:05.533075 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.533042    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp"]
Oct 30 03:58:05.731319 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.731225    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85bf77f655-fg6hp" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:58:05.928744 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:05.928710    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" path="/var/lib/kubelet/pods/cee40dd7-e512-4a34-8fa3-2680d072a4f0/volumes"
Oct 30 03:58:12.058034 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:12.057992    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:58:22.057834 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:22.057783    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:58:24.081922 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:58:24.081862    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:58:32.058141 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:32.058082    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:58:42.057815 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:42.057774    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:58:44.107203 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107166    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"]
Oct 30 03:58:44.107731 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107601    4494 cpu_manager.go:414] "containerMap: removing container" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="storage-initializer"
Oct 30 03:58:44.107731 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107617    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="storage-initializer"
Oct 30 03:58:44.107731 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107638    4494 cpu_manager.go:414] "containerMap: removing container" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy"
Oct 30 03:58:44.107731 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107644    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy"
Oct 30 03:58:44.107731 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107666    4494 cpu_manager.go:414] "containerMap: removing container" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main"
Oct 30 03:58:44.107731 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107674    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main"
Oct 30 03:58:44.108079 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107745    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy"
Oct 30 03:58:44.108079 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107759    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee40dd7-e512-4a34-8fa3-2680d072a4f0" containerName="main"
Oct 30 03:58:44.108079 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107822    4494 cpu_manager.go:414] "containerMap: removing container" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy"
Oct 30 03:58:44.108079 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107828    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy"
Oct 30 03:58:44.108079 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.107893    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a6f815f-53db-4f8e-bb34-0feff9ebd5ce" containerName="istio-proxy"
Oct 30 03:58:44.111049 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.111025    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.144364 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.144326    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:58:44.144555 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.144365    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Oct 30 03:58:44.144555 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.144382    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Oct 30 03:58:44.144555 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.144381    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Oct 30 03:58:44.144555 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.144443    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Oct 30 03:58:44.144555 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.144464    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-9ppkf\""
Oct 30 03:58:44.145946 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.145924    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"]
Oct 30 03:58:44.300000 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.299958    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a36977-57da-4752-b2bf-8a31a4a78d29-tls-certs\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.300177 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.300010    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-model-cache\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.300177 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.300088    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-tmp-dir\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.300177 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.300110    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-home\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.300177 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.300130    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.300323 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.300219    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56a36977-57da-4752-b2bf-8a31a4a78d29-cabundle-cert\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.300323 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.300242    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-dshm\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.300323 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.300307    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fwn\" (UniqueName: \"kubernetes.io/projected/56a36977-57da-4752-b2bf-8a31a4a78d29-kube-api-access-86fwn\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401097 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401002    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-model-cache\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401097 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401048    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-tmp-dir\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401097 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401066    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-home\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401097 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401085    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401423 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401122    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56a36977-57da-4752-b2bf-8a31a4a78d29-cabundle-cert\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401423 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401148    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-dshm\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401423 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401212    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fwn\" (UniqueName: \"kubernetes.io/projected/56a36977-57da-4752-b2bf-8a31a4a78d29-kube-api-access-86fwn\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401423 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401258    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a36977-57da-4752-b2bf-8a31a4a78d29-tls-certs\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401609 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401522    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-model-cache\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401609 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401556    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-tmp-dir\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401725 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401712    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-home\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.401777 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.401766    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.402141 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.402118    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56a36977-57da-4752-b2bf-8a31a4a78d29-cabundle-cert\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.404284 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.404255    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-dshm\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.404475 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.404456    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a36977-57da-4752-b2bf-8a31a4a78d29-tls-certs\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.424775 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.424745    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fwn\" (UniqueName: \"kubernetes.io/projected/56a36977-57da-4752-b2bf-8a31a4a78d29-kube-api-access-86fwn\") pod \"lora-single-adapter-test-kserve-8464d4bf5-v54t8\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.429458 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.429430    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:44.749728 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:44.749701    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"]
Oct 30 03:58:44.750803 ip-10-0-130-130 kubenswrapper[4494]: W1030 03:58:44.750776    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a36977_57da_4752_b2bf_8a31a4a78d29.slice/crio-9f25a08f919ecfd6232aae05ccdfa627b9074b1fa9a4c51fef8751acfe2097af WatchSource:0}: Error finding container 9f25a08f919ecfd6232aae05ccdfa627b9074b1fa9a4c51fef8751acfe2097af: Status 404 returned error can't find the container with id 9f25a08f919ecfd6232aae05ccdfa627b9074b1fa9a4c51fef8751acfe2097af
Oct 30 03:58:45.658775 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:45.658720    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" event={"ID":"56a36977-57da-4752-b2bf-8a31a4a78d29","Type":"ContainerStarted","Data":"5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0"}
Oct 30 03:58:45.659520 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:45.659480    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" event={"ID":"56a36977-57da-4752-b2bf-8a31a4a78d29","Type":"ContainerStarted","Data":"9f25a08f919ecfd6232aae05ccdfa627b9074b1fa9a4c51fef8751acfe2097af"}
Oct 30 03:58:48.675432 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:48.675399    4494 generic.go:356] "Generic (PLEG): container finished" podID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerID="5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0" exitCode=0
Oct 30 03:58:48.675945 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:48.675476    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" event={"ID":"56a36977-57da-4752-b2bf-8a31a4a78d29","Type":"ContainerDied","Data":"5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0"}
Oct 30 03:58:49.682384 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:49.682342    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" event={"ID":"56a36977-57da-4752-b2bf-8a31a4a78d29","Type":"ContainerStarted","Data":"6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c"}
Oct 30 03:58:49.754584 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:49.754517    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" podStartSLOduration=2.87687745 podStartE2EDuration="5.754500672s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.877623222s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:49.741460223 +0000 UTC m=+3793.947640443" watchObservedRunningTime="2026-10-30 03:58:49.754500672 +0000 UTC m=+3793.960680979"
Oct 30 03:58:50.430079 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:50.430040    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:52.057744 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:52.057690    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:58:54.083870 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:58:54.083834    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:58:54.430733 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:54.430636    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 03:58:54.432276 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:58:54.432244    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Oct 30 03:59:02.057967 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:02.057919    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:59:04.430508 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:04.430465    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Oct 30 03:59:12.058281 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:12.058231    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:59:14.430041 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:14.429994    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Oct 30 03:59:22.058457 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:22.058413    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Oct 30 03:59:24.083005 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:59:24.082931    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:59:24.430744 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:24.430636    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Oct 30 03:59:32.067210 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:32.067173    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:59:32.074909 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:32.074875    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:59:34.430405 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:34.430352    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Oct 30 03:59:42.059479 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:42.059432    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"]
Oct 30 03:59:44.430713 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:44.430663    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Oct 30 03:59:54.082269 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:59:54.082226    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 03:59:54.430554 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:54.430448    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Oct 30 03:59:57.083914 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.083835    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main" containerID="cri-o://5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4" gracePeriod=45
Oct 30 03:59:57.843387 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.843360    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:59:57.900111 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900008    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95644d78-79a3-48e4-8035-6905dd99d92d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95644d78-79a3-48e4-8035-6905dd99d92d-cabundle-cert\") pod \"95644d78-79a3-48e4-8035-6905dd99d92d\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") "
Oct 30 03:59:57.900288 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900114    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-kserve-provision-location\") pod \"95644d78-79a3-48e4-8035-6905dd99d92d\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") "
Oct 30 03:59:57.900288 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900166    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-home\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-home\") pod \"95644d78-79a3-48e4-8035-6905dd99d92d\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") "
Oct 30 03:59:57.900288 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900187    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-model-cache\") pod \"95644d78-79a3-48e4-8035-6905dd99d92d\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") "
Oct 30 03:59:57.900288 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900218    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95644d78-79a3-48e4-8035-6905dd99d92d-tls-certs\" (UniqueName: \"kubernetes.io/secret/95644d78-79a3-48e4-8035-6905dd99d92d-tls-certs\") pod \"95644d78-79a3-48e4-8035-6905dd99d92d\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") "
Oct 30 03:59:57.900288 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900245    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-tmp-dir\") pod \"95644d78-79a3-48e4-8035-6905dd99d92d\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") "
Oct 30 03:59:57.900288 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900276    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-dshm\") pod \"95644d78-79a3-48e4-8035-6905dd99d92d\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") "
Oct 30 03:59:57.900607 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900307    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95644d78-79a3-48e4-8035-6905dd99d92d-kube-api-access-xx5cr\" (UniqueName: \"kubernetes.io/projected/95644d78-79a3-48e4-8035-6905dd99d92d-kube-api-access-xx5cr\") pod \"95644d78-79a3-48e4-8035-6905dd99d92d\" (UID: \"95644d78-79a3-48e4-8035-6905dd99d92d\") "
Oct 30 03:59:57.900607 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900491    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-model-cache" pod "95644d78-79a3-48e4-8035-6905dd99d92d" (UID: "95644d78-79a3-48e4-8035-6905dd99d92d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:57.900607 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900447    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95644d78-79a3-48e4-8035-6905dd99d92d-cabundle-cert" pod "95644d78-79a3-48e4-8035-6905dd99d92d" (UID: "95644d78-79a3-48e4-8035-6905dd99d92d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:57.900979 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.900950    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-home" pod "95644d78-79a3-48e4-8035-6905dd99d92d" (UID: "95644d78-79a3-48e4-8035-6905dd99d92d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:57.903934 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.903883    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95644d78-79a3-48e4-8035-6905dd99d92d-tls-certs" pod "95644d78-79a3-48e4-8035-6905dd99d92d" (UID: "95644d78-79a3-48e4-8035-6905dd99d92d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:59:57.904055 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.903955    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-dshm" pod "95644d78-79a3-48e4-8035-6905dd99d92d" (UID: "95644d78-79a3-48e4-8035-6905dd99d92d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:57.904055 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.904009    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95644d78-79a3-48e4-8035-6905dd99d92d-kube-api-access-xx5cr" pod "95644d78-79a3-48e4-8035-6905dd99d92d" (UID: "95644d78-79a3-48e4-8035-6905dd99d92d"). InnerVolumeSpecName "kube-api-access-xx5cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:59:57.913569 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.913534    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-tmp-dir" pod "95644d78-79a3-48e4-8035-6905dd99d92d" (UID: "95644d78-79a3-48e4-8035-6905dd99d92d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:57.922075 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:57.922043    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-kserve-provision-location" pod "95644d78-79a3-48e4-8035-6905dd99d92d" (UID: "95644d78-79a3-48e4-8035-6905dd99d92d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:58.001304 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.001259    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.001304 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.001289    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx5cr\" (UniqueName: \"kubernetes.io/projected/95644d78-79a3-48e4-8035-6905dd99d92d-kube-api-access-xx5cr\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.001304 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.001300    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95644d78-79a3-48e4-8035-6905dd99d92d-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.001304 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.001309    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.001304 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.001318    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.001714 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.001325    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.001714 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.001333    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95644d78-79a3-48e4-8035-6905dd99d92d-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.001714 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.001340    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95644d78-79a3-48e4-8035-6905dd99d92d-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 03:59:58.022834 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.022800    4494 generic.go:356] "Generic (PLEG): container finished" podID="95644d78-79a3-48e4-8035-6905dd99d92d" containerID="5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4" exitCode=0
Oct 30 03:59:58.023067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.022853    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"
Oct 30 03:59:58.023067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.022873    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" event={"ID":"95644d78-79a3-48e4-8035-6905dd99d92d","Type":"ContainerDied","Data":"5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4"}
Oct 30 03:59:58.023067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.022947    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp" event={"ID":"95644d78-79a3-48e4-8035-6905dd99d92d","Type":"ContainerDied","Data":"80f564167a5b24b164cddcd999a41d38bb6db1bc2e7070a951c9d04650ee1110"}
Oct 30 03:59:58.023067 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.022978    4494 scope.go:122] "RemoveContainer" containerID="5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4"
Oct 30 03:59:58.031660 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.031637    4494 scope.go:122] "RemoveContainer" containerID="117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5"
Oct 30 03:59:58.039583 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.039558    4494 scope.go:122] "RemoveContainer" containerID="5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4"
Oct 30 03:59:58.039883 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:59:58.039856    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4\": container with ID starting with 5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4 not found: ID does not exist" containerID="5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4"
Oct 30 03:59:58.039949 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.039912    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4"} err="failed to get container status \"5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4\": rpc error: code = NotFound desc = could not find container \"5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4\": container with ID starting with 5a957c9802751f4495afb07421fb2f726753bfdac8ec3e991f1e8c58050a1ab4 not found: ID does not exist"
Oct 30 03:59:58.039949 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.039938    4494 scope.go:122] "RemoveContainer" containerID="117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5"
Oct 30 03:59:58.040225 ip-10-0-130-130 kubenswrapper[4494]: E1030 03:59:58.040207    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5\": container with ID starting with 117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5 not found: ID does not exist" containerID="117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5"
Oct 30 03:59:58.040273 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.040229    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5"} err="failed to get container status \"117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5\": rpc error: code = NotFound desc = could not find container \"117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5\": container with ID starting with 117fa53b4ecc40da2952d21b622adfb4db0b2a6bceaee069f98532aa14459ce5 not found: ID does not exist"
Oct 30 03:59:58.075092 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.075050    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"]
Oct 30 03:59:58.081894 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:58.081863    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-585t8fp"]
Oct 30 03:59:59.926571 ip-10-0-130-130 kubenswrapper[4494]: I1030 03:59:59.926523    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" path="/var/lib/kubelet/pods/95644d78-79a3-48e4-8035-6905dd99d92d/volumes"
Oct 30 04:00:04.440477 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:04.440438    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 04:00:04.448473 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:04.448440    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 04:00:24.083623 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:00:24.083575    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:00:30.675654 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:30.675615    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-8464d4bf5-v54t8_56a36977-57da-4752-b2bf-8a31a4a78d29/storage-initializer/0.log"
Oct 30 04:00:30.716392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:30.716362    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-8464d4bf5-v54t8_56a36977-57da-4752-b2bf-8a31a4a78d29/main/0.log"
Oct 30 04:00:35.474287 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.474246    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"]
Oct 30 04:00:35.536504 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.536442    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"]
Oct 30 04:00:35.537055 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.537030    4494 cpu_manager.go:414] "containerMap: removing container" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main"
Oct 30 04:00:35.537055 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.537057    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main"
Oct 30 04:00:35.537238 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.537078    4494 cpu_manager.go:414] "containerMap: removing container" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="storage-initializer"
Oct 30 04:00:35.537238 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.537086    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="storage-initializer"
Oct 30 04:00:35.537238 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.537217    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="95644d78-79a3-48e4-8035-6905dd99d92d" containerName="main"
Oct 30 04:00:35.542133 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.542113    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.565856 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.565821    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"]
Oct 30 04:00:35.612486 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.612454    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Oct 30 04:00:35.612679 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.612507    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-b4mk4\""
Oct 30 04:00:35.612679 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.612507    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Oct 30 04:00:35.628714 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.628683    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:00:35.628714 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.628690    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Oct 30 04:00:35.628714 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.628711    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Oct 30 04:00:35.647141 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.647110    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t692\" (UniqueName: \"kubernetes.io/projected/a5d56520-7b23-4b1f-ad90-242116bc3de0-kube-api-access-7t692\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.647286 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.647160    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.647286 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.647184    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.647406 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.647350    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.647406 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.647396    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.647479 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.647456    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5d56520-7b23-4b1f-ad90-242116bc3de0-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.647516 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.647488    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.647516 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.647507    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5d56520-7b23-4b1f-ad90-242116bc3de0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.748492 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.748395    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.748492 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.748470    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.748728 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.748497    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.748728 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.748520    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5d56520-7b23-4b1f-ad90-242116bc3de0-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.748728 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.748554    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.748728 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.748578    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5d56520-7b23-4b1f-ad90-242116bc3de0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.748728 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.748625    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t692\" (UniqueName: \"kubernetes.io/projected/a5d56520-7b23-4b1f-ad90-242116bc3de0-kube-api-access-7t692\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.748728 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.748672    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.749080 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.748928    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.749080 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.748967    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.749319 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.749297    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.749377 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.749359    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5d56520-7b23-4b1f-ad90-242116bc3de0-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.749473 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.749453    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.751065 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.751033    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.751289 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.751267    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5d56520-7b23-4b1f-ad90-242116bc3de0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.789533 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.789486    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t692\" (UniqueName: \"kubernetes.io/projected/a5d56520-7b23-4b1f-ad90-242116bc3de0-kube-api-access-7t692\") pod \"custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:35.860636 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:35.860594    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:36.186450 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:36.186418    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"]
Oct 30 04:00:36.188446 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:00:36.188414    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d56520_7b23_4b1f_ad90_242116bc3de0.slice/crio-6d83f63eeeded2f65d05b914569b8e6e3b5ec0f9d6f302efe75ef48ec7b0215d WatchSource:0}: Error finding container 6d83f63eeeded2f65d05b914569b8e6e3b5ec0f9d6f302efe75ef48ec7b0215d: Status 404 returned error can't find the container with id 6d83f63eeeded2f65d05b914569b8e6e3b5ec0f9d6f302efe75ef48ec7b0215d
Oct 30 04:00:36.198192 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:36.198148    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" event={"ID":"a5d56520-7b23-4b1f-ad90-242116bc3de0","Type":"ContainerStarted","Data":"6d83f63eeeded2f65d05b914569b8e6e3b5ec0f9d6f302efe75ef48ec7b0215d"}
Oct 30 04:00:36.204123 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:36.204104    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Oct 30 04:00:37.206091 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:37.206056    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" event={"ID":"a5d56520-7b23-4b1f-ad90-242116bc3de0","Type":"ContainerStarted","Data":"8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e"}
Oct 30 04:00:39.216617 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:39.216584    4494 generic.go:356] "Generic (PLEG): container finished" podID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerID="8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e" exitCode=0
Oct 30 04:00:39.217023 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:39.216641    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" event={"ID":"a5d56520-7b23-4b1f-ad90-242116bc3de0","Type":"ContainerDied","Data":"8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e"}
Oct 30 04:00:40.223858 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:40.223811    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" event={"ID":"a5d56520-7b23-4b1f-ad90-242116bc3de0","Type":"ContainerStarted","Data":"26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3"}
Oct 30 04:00:40.296150 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:40.296085    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podStartSLOduration=3.078321976 podStartE2EDuration="5.296066079s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.217744103s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:40.283413723 +0000 UTC m=+3904.489593952" watchObservedRunningTime="2026-10-30 04:00:40.296066079 +0000 UTC m=+3904.502246298"
Oct 30 04:00:45.861000 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:45.860959    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:45.861404 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:45.861030    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:00:45.862492 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:45.862462    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:00:50.498562 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:50.498454    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main" containerID="cri-o://6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c" gracePeriod=45
Oct 30 04:00:51.242017 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.241984    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 04:00:51.276833 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.276796    4494 generic.go:356] "Generic (PLEG): container finished" podID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerID="6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c" exitCode=0
Oct 30 04:00:51.277071 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.276846    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" event={"ID":"56a36977-57da-4752-b2bf-8a31a4a78d29","Type":"ContainerDied","Data":"6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c"}
Oct 30 04:00:51.277071 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.276881    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"
Oct 30 04:00:51.277071 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.276921    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8" event={"ID":"56a36977-57da-4752-b2bf-8a31a4a78d29","Type":"ContainerDied","Data":"9f25a08f919ecfd6232aae05ccdfa627b9074b1fa9a4c51fef8751acfe2097af"}
Oct 30 04:00:51.277071 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.276950    4494 scope.go:122] "RemoveContainer" containerID="6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c"
Oct 30 04:00:51.287544 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.287518    4494 scope.go:122] "RemoveContainer" containerID="5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0"
Oct 30 04:00:51.297183 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.297153    4494 scope.go:122] "RemoveContainer" containerID="6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c"
Oct 30 04:00:51.297631 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:00:51.297604    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c\": container with ID starting with 6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c not found: ID does not exist" containerID="6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c"
Oct 30 04:00:51.297754 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.297641    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c"} err="failed to get container status \"6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c\": rpc error: code = NotFound desc = could not find container \"6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c\": container with ID starting with 6dda31ea36dfb97a5d9e27bb0736a6689b115d796801a62b2430fb23c0c3af0c not found: ID does not exist"
Oct 30 04:00:51.297754 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.297662    4494 scope.go:122] "RemoveContainer" containerID="5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0"
Oct 30 04:00:51.298032 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:00:51.298010    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0\": container with ID starting with 5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0 not found: ID does not exist" containerID="5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0"
Oct 30 04:00:51.298123 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.298037    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0"} err="failed to get container status \"5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0\": rpc error: code = NotFound desc = could not find container \"5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0\": container with ID starting with 5574a1363bcbddbd8f8a9481410c09a79ae2ddb87a0015a53d13366ecfa29aa0 not found: ID does not exist"
Oct 30 04:00:51.396003 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.395963    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-home\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-home\") pod \"56a36977-57da-4752-b2bf-8a31a4a78d29\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") "
Oct 30 04:00:51.396176 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396035    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a36977-57da-4752-b2bf-8a31a4a78d29-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a36977-57da-4752-b2bf-8a31a4a78d29-tls-certs\") pod \"56a36977-57da-4752-b2bf-8a31a4a78d29\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") "
Oct 30 04:00:51.396176 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396073    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-model-cache\") pod \"56a36977-57da-4752-b2bf-8a31a4a78d29\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") "
Oct 30 04:00:51.396176 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396124    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a36977-57da-4752-b2bf-8a31a4a78d29-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56a36977-57da-4752-b2bf-8a31a4a78d29-cabundle-cert\") pod \"56a36977-57da-4752-b2bf-8a31a4a78d29\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") "
Oct 30 04:00:51.396321 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396187    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a36977-57da-4752-b2bf-8a31a4a78d29-kube-api-access-86fwn\" (UniqueName: \"kubernetes.io/projected/56a36977-57da-4752-b2bf-8a31a4a78d29-kube-api-access-86fwn\") pod \"56a36977-57da-4752-b2bf-8a31a4a78d29\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") "
Oct 30 04:00:51.396321 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396224    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-tmp-dir\") pod \"56a36977-57da-4752-b2bf-8a31a4a78d29\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") "
Oct 30 04:00:51.396321 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396302    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-dshm\") pod \"56a36977-57da-4752-b2bf-8a31a4a78d29\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") "
Oct 30 04:00:51.396478 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396341    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-kserve-provision-location\") pod \"56a36977-57da-4752-b2bf-8a31a4a78d29\" (UID: \"56a36977-57da-4752-b2bf-8a31a4a78d29\") "
Oct 30 04:00:51.396478 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396374    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-model-cache" pod "56a36977-57da-4752-b2bf-8a31a4a78d29" (UID: "56a36977-57da-4752-b2bf-8a31a4a78d29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:51.396478 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396415    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-home" pod "56a36977-57da-4752-b2bf-8a31a4a78d29" (UID: "56a36977-57da-4752-b2bf-8a31a4a78d29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:51.396654 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396626    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a36977-57da-4752-b2bf-8a31a4a78d29-cabundle-cert" pod "56a36977-57da-4752-b2bf-8a31a4a78d29" (UID: "56a36977-57da-4752-b2bf-8a31a4a78d29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:00:51.396790 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396771    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:00:51.396859 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396798    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56a36977-57da-4752-b2bf-8a31a4a78d29-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:00:51.396859 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.396812    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:00:51.398807 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.398782    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a36977-57da-4752-b2bf-8a31a4a78d29-tls-certs" pod "56a36977-57da-4752-b2bf-8a31a4a78d29" (UID: "56a36977-57da-4752-b2bf-8a31a4a78d29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:00:51.399437 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.399148    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-dshm" pod "56a36977-57da-4752-b2bf-8a31a4a78d29" (UID: "56a36977-57da-4752-b2bf-8a31a4a78d29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:51.399529 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.399506    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a36977-57da-4752-b2bf-8a31a4a78d29-kube-api-access-86fwn" pod "56a36977-57da-4752-b2bf-8a31a4a78d29" (UID: "56a36977-57da-4752-b2bf-8a31a4a78d29"). InnerVolumeSpecName "kube-api-access-86fwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:00:51.410688 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.410654    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-tmp-dir" pod "56a36977-57da-4752-b2bf-8a31a4a78d29" (UID: "56a36977-57da-4752-b2bf-8a31a4a78d29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:51.420024 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.419991    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-kserve-provision-location" pod "56a36977-57da-4752-b2bf-8a31a4a78d29" (UID: "56a36977-57da-4752-b2bf-8a31a4a78d29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:00:51.498114 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.498067    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:00:51.498114 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.498106    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:00:51.498114 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.498121    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a36977-57da-4752-b2bf-8a31a4a78d29-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:00:51.498374 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.498136    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86fwn\" (UniqueName: \"kubernetes.io/projected/56a36977-57da-4752-b2bf-8a31a4a78d29-kube-api-access-86fwn\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:00:51.498374 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.498148    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a36977-57da-4752-b2bf-8a31a4a78d29-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:00:51.623683 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.623646    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"]
Oct 30 04:00:51.635470 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.635433    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-8464d4bf5-v54t8"]
Oct 30 04:00:51.932191 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:51.932099    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" path="/var/lib/kubelet/pods/56a36977-57da-4752-b2bf-8a31a4a78d29/volumes"
Oct 30 04:00:54.082985 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:00:54.082943    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:00:55.861236 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:55.861193    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:00:58.128737 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.128701    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"]
Oct 30 04:00:58.129217 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.129118    4494 cpu_manager.go:414] "containerMap: removing container" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="storage-initializer"
Oct 30 04:00:58.129217 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.129135    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="storage-initializer"
Oct 30 04:00:58.129217 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.129143    4494 cpu_manager.go:414] "containerMap: removing container" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main"
Oct 30 04:00:58.129217 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.129148    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main"
Oct 30 04:00:58.129217 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.129212    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a36977-57da-4752-b2bf-8a31a4a78d29" containerName="main"
Oct 30 04:00:58.133281 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.133261    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.155997 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.155972    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:00:58.156172 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.156150    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Oct 30 04:00:58.156274 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.156254    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Oct 30 04:00:58.170268 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.170232    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"]
Oct 30 04:00:58.189068 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.189034    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Oct 30 04:00:58.189270 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.189148    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-dgtkc\""
Oct 30 04:00:58.189270 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.189247    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Oct 30 04:00:58.261262 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.261223    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29995b41-19f0-402f-9f23-604c2f61acd7-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.261262 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.261267    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.261524 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.261298    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-dshm\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.261524 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.261363    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-home\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.261524 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.261465    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.261524 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.261488    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5j8c\" (UniqueName: \"kubernetes.io/projected/29995b41-19f0-402f-9f23-604c2f61acd7-kube-api-access-m5j8c\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.261524 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.261512    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-model-cache\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.261674 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.261555    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29995b41-19f0-402f-9f23-604c2f61acd7-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.362973 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.362925    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29995b41-19f0-402f-9f23-604c2f61acd7-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363167 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363001    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29995b41-19f0-402f-9f23-604c2f61acd7-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363167 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363033    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363167 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363073    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-dshm\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363167 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363106    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-home\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363167 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363146    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363466 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363173    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5j8c\" (UniqueName: \"kubernetes.io/projected/29995b41-19f0-402f-9f23-604c2f61acd7-kube-api-access-m5j8c\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363466 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363209    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-model-cache\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363572 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363514    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-model-cache\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363626 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363574    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363670 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363631    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-home\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363790 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363771    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.363838 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.363797    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29995b41-19f0-402f-9f23-604c2f61acd7-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.365970 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.365945    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29995b41-19f0-402f-9f23-604c2f61acd7-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.366088 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.365967    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-dshm\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.387939 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.387843    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5j8c\" (UniqueName: \"kubernetes.io/projected/29995b41-19f0-402f-9f23-604c2f61acd7-kube-api-access-m5j8c\") pod \"lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.448028 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.447984    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:00:58.775285 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:58.775256    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"]
Oct 30 04:00:58.776699 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:00:58.776659    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29995b41_19f0_402f_9f23_604c2f61acd7.slice/crio-5052ebc55d5dc3176b5803c020284d5cd9ddaf123d717811a78ae76dd921d743 WatchSource:0}: Error finding container 5052ebc55d5dc3176b5803c020284d5cd9ddaf123d717811a78ae76dd921d743: Status 404 returned error can't find the container with id 5052ebc55d5dc3176b5803c020284d5cd9ddaf123d717811a78ae76dd921d743
Oct 30 04:00:59.314709 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:59.314669    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" event={"ID":"29995b41-19f0-402f-9f23-604c2f61acd7","Type":"ContainerStarted","Data":"29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2"}
Oct 30 04:00:59.315173 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:00:59.314733    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" event={"ID":"29995b41-19f0-402f-9f23-604c2f61acd7","Type":"ContainerStarted","Data":"5052ebc55d5dc3176b5803c020284d5cd9ddaf123d717811a78ae76dd921d743"}
Oct 30 04:01:02.330710 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:02.330676    4494 generic.go:356] "Generic (PLEG): container finished" podID="29995b41-19f0-402f-9f23-604c2f61acd7" containerID="29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2" exitCode=0
Oct 30 04:01:02.331171 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:02.330765    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" event={"ID":"29995b41-19f0-402f-9f23-604c2f61acd7","Type":"ContainerDied","Data":"29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2"}
Oct 30 04:01:03.338779 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:03.338738    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" event={"ID":"29995b41-19f0-402f-9f23-604c2f61acd7","Type":"ContainerStarted","Data":"0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410"}
Oct 30 04:01:03.421651 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:03.421586    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podStartSLOduration=2.322738024 podStartE2EDuration="5.421571043s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.098833019s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:03.420092805 +0000 UTC m=+3927.626273034" watchObservedRunningTime="2026-10-30 04:01:03.421571043 +0000 UTC m=+3927.627751261"
Oct 30 04:01:03.449079 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:03.449041    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:01:05.862032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:05.861969    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:01:08.448472 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:08.448428    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:01:08.450278 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:08.450243    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Oct 30 04:01:15.861443 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:15.861397    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:01:18.451825 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:18.451759    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Oct 30 04:01:24.088686 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:01:24.088643    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:01:25.861615 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:25.861560    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:01:28.449374 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:28.449319    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Oct 30 04:01:35.861779 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:35.861715    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:01:38.448431 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:38.448385    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Oct 30 04:01:45.861067 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:45.861015    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:01:48.449547 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:48.449496    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Oct 30 04:01:54.084644 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:01:54.084595    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:01:55.862047 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:55.861989    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:01:58.448735 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:01:58.448678    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Oct 30 04:02:05.861951 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:05.861878    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:02:08.448999 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:08.448947    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Oct 30 04:02:15.862031 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:15.861977    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:02:18.448573 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:18.448463    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Oct 30 04:02:24.084454 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:02:24.084414    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:02:25.861536 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:25.861483    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:02:28.459203 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:28.459169    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:02:28.472047 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:28.472011    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:02:35.861428 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:35.861373    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Oct 30 04:02:45.872511 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:45.872481    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:02:45.880275 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:45.880241    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:02:54.085095 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:02:54.085049    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:02:58.700936 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:02:58.700876    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"]
Oct 30 04:03:06.578483 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:06.578448    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm_29995b41-19f0-402f-9f23-604c2f61acd7/storage-initializer/0.log"
Oct 30 04:03:06.622745 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:06.622705    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm_29995b41-19f0-402f-9f23-604c2f61acd7/main/0.log"
Oct 30 04:03:10.200984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:10.200945    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"]
Oct 30 04:03:13.724832 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:13.724774    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main" containerID="cri-o://26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3" gracePeriod=45
Oct 30 04:03:14.476041 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.476012    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:03:14.564716 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.564683    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-model-cache\") pod \"a5d56520-7b23-4b1f-ad90-242116bc3de0\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") "
Oct 30 04:03:14.564919 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.564747    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d56520-7b23-4b1f-ad90-242116bc3de0-kube-api-access-7t692\" (UniqueName: \"kubernetes.io/projected/a5d56520-7b23-4b1f-ad90-242116bc3de0-kube-api-access-7t692\") pod \"a5d56520-7b23-4b1f-ad90-242116bc3de0\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") "
Oct 30 04:03:14.564919 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.564782    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d56520-7b23-4b1f-ad90-242116bc3de0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5d56520-7b23-4b1f-ad90-242116bc3de0-cabundle-cert\") pod \"a5d56520-7b23-4b1f-ad90-242116bc3de0\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") "
Oct 30 04:03:14.564919 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.564827    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-home\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-home\") pod \"a5d56520-7b23-4b1f-ad90-242116bc3de0\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") "
Oct 30 04:03:14.564919 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.564915    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-kserve-provision-location\") pod \"a5d56520-7b23-4b1f-ad90-242116bc3de0\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") "
Oct 30 04:03:14.565151 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.564974    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-tmp-dir\") pod \"a5d56520-7b23-4b1f-ad90-242116bc3de0\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") "
Oct 30 04:03:14.565151 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.564999    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-dshm\") pod \"a5d56520-7b23-4b1f-ad90-242116bc3de0\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") "
Oct 30 04:03:14.565151 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.565025    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5d56520-7b23-4b1f-ad90-242116bc3de0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5d56520-7b23-4b1f-ad90-242116bc3de0-tls-certs\") pod \"a5d56520-7b23-4b1f-ad90-242116bc3de0\" (UID: \"a5d56520-7b23-4b1f-ad90-242116bc3de0\") "
Oct 30 04:03:14.565151 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.565068    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-model-cache" pod "a5d56520-7b23-4b1f-ad90-242116bc3de0" (UID: "a5d56520-7b23-4b1f-ad90-242116bc3de0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:14.565357 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.565308    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.565414 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.565365    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d56520-7b23-4b1f-ad90-242116bc3de0-cabundle-cert" pod "a5d56520-7b23-4b1f-ad90-242116bc3de0" (UID: "a5d56520-7b23-4b1f-ad90-242116bc3de0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:14.565679 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.565648    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-home" pod "a5d56520-7b23-4b1f-ad90-242116bc3de0" (UID: "a5d56520-7b23-4b1f-ad90-242116bc3de0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:14.567639 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.567602    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d56520-7b23-4b1f-ad90-242116bc3de0-kube-api-access-7t692" pod "a5d56520-7b23-4b1f-ad90-242116bc3de0" (UID: "a5d56520-7b23-4b1f-ad90-242116bc3de0"). InnerVolumeSpecName "kube-api-access-7t692". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:14.567773 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.567725    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d56520-7b23-4b1f-ad90-242116bc3de0-tls-certs" pod "a5d56520-7b23-4b1f-ad90-242116bc3de0" (UID: "a5d56520-7b23-4b1f-ad90-242116bc3de0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:14.568167 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.568143    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-dshm" pod "a5d56520-7b23-4b1f-ad90-242116bc3de0" (UID: "a5d56520-7b23-4b1f-ad90-242116bc3de0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:14.577249 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.577216    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-tmp-dir" pod "a5d56520-7b23-4b1f-ad90-242116bc3de0" (UID: "a5d56520-7b23-4b1f-ad90-242116bc3de0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:14.590944 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.590872    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-kserve-provision-location" pod "a5d56520-7b23-4b1f-ad90-242116bc3de0" (UID: "a5d56520-7b23-4b1f-ad90-242116bc3de0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:14.666094 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.666049    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.666094 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.666087    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.666369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.666103    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.666369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.666120    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5d56520-7b23-4b1f-ad90-242116bc3de0-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.666369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.666129    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5d56520-7b23-4b1f-ad90-242116bc3de0-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.666369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.666137    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t692\" (UniqueName: \"kubernetes.io/projected/a5d56520-7b23-4b1f-ad90-242116bc3de0-kube-api-access-7t692\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.666369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.666145    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5d56520-7b23-4b1f-ad90-242116bc3de0-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:14.993547 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.993459    4494 generic.go:356] "Generic (PLEG): container finished" podID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerID="26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3" exitCode=0
Oct 30 04:03:14.993547 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.993512    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"
Oct 30 04:03:14.994075 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.993505    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" event={"ID":"a5d56520-7b23-4b1f-ad90-242116bc3de0","Type":"ContainerDied","Data":"26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3"}
Oct 30 04:03:14.994075 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.993634    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk" event={"ID":"a5d56520-7b23-4b1f-ad90-242116bc3de0","Type":"ContainerDied","Data":"6d83f63eeeded2f65d05b914569b8e6e3b5ec0f9d6f302efe75ef48ec7b0215d"}
Oct 30 04:03:14.994075 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:14.993657    4494 scope.go:122] "RemoveContainer" containerID="26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3"
Oct 30 04:03:15.006337 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:15.006317    4494 scope.go:122] "RemoveContainer" containerID="8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e"
Oct 30 04:03:15.015483 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:15.015462    4494 scope.go:122] "RemoveContainer" containerID="26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3"
Oct 30 04:03:15.015801 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:03:15.015779    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3\": container with ID starting with 26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3 not found: ID does not exist" containerID="26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3"
Oct 30 04:03:15.015875 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:15.015815    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3"} err="failed to get container status \"26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3\": rpc error: code = NotFound desc = could not find container \"26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3\": container with ID starting with 26ed04c31da1cc7c8c23e6df29998ada10319f15e819176fe014d71639ad21e3 not found: ID does not exist"
Oct 30 04:03:15.015875 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:15.015842    4494 scope.go:122] "RemoveContainer" containerID="8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e"
Oct 30 04:03:15.016141 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:03:15.016120    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e\": container with ID starting with 8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e not found: ID does not exist" containerID="8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e"
Oct 30 04:03:15.016196 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:15.016154    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e"} err="failed to get container status \"8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e\": rpc error: code = NotFound desc = could not find container \"8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e\": container with ID starting with 8ba9e5a3ad0bafe17fe39ee4d0fa498302c6792a0b9dae919ff600d26fdcb39e not found: ID does not exist"
Oct 30 04:03:15.058653 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:15.058612    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"]
Oct 30 04:03:15.070673 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:15.070639    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7766bf86c6-qwwhk"]
Oct 30 04:03:15.926195 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:15.926162    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" path="/var/lib/kubelet/pods/a5d56520-7b23-4b1f-ad90-242116bc3de0/volumes"
Oct 30 04:03:24.082644 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:03:24.082598    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:03:25.225566 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.225521    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main" containerID="cri-o://0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410" gracePeriod=45
Oct 30 04:03:25.460343 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.460293    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Oct 30 04:03:25.876540 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.876514    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:03:25.965688 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.965609    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29995b41-19f0-402f-9f23-604c2f61acd7-kube-api-access-m5j8c\" (UniqueName: \"kubernetes.io/projected/29995b41-19f0-402f-9f23-604c2f61acd7-kube-api-access-m5j8c\") pod \"29995b41-19f0-402f-9f23-604c2f61acd7\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") "
Oct 30 04:03:25.965688 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.965650    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-tmp-dir\") pod \"29995b41-19f0-402f-9f23-604c2f61acd7\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") "
Oct 30 04:03:25.965688 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.965670    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29995b41-19f0-402f-9f23-604c2f61acd7-tls-certs\" (UniqueName: \"kubernetes.io/secret/29995b41-19f0-402f-9f23-604c2f61acd7-tls-certs\") pod \"29995b41-19f0-402f-9f23-604c2f61acd7\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") "
Oct 30 04:03:25.965992 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.965699    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29995b41-19f0-402f-9f23-604c2f61acd7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29995b41-19f0-402f-9f23-604c2f61acd7-cabundle-cert\") pod \"29995b41-19f0-402f-9f23-604c2f61acd7\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") "
Oct 30 04:03:25.965992 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.965785    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-model-cache\") pod \"29995b41-19f0-402f-9f23-604c2f61acd7\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") "
Oct 30 04:03:25.965992 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.965821    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-home\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-home\") pod \"29995b41-19f0-402f-9f23-604c2f61acd7\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") "
Oct 30 04:03:25.965992 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.965857    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-dshm\") pod \"29995b41-19f0-402f-9f23-604c2f61acd7\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") "
Oct 30 04:03:25.965992 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.965935    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-kserve-provision-location\") pod \"29995b41-19f0-402f-9f23-604c2f61acd7\" (UID: \"29995b41-19f0-402f-9f23-604c2f61acd7\") "
Oct 30 04:03:25.966265 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.966147    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29995b41-19f0-402f-9f23-604c2f61acd7-cabundle-cert" pod "29995b41-19f0-402f-9f23-604c2f61acd7" (UID: "29995b41-19f0-402f-9f23-604c2f61acd7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:25.966328 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.966314    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29995b41-19f0-402f-9f23-604c2f61acd7-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:25.966555 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.966413    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-home" pod "29995b41-19f0-402f-9f23-604c2f61acd7" (UID: "29995b41-19f0-402f-9f23-604c2f61acd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:25.966728 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.966647    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-model-cache" pod "29995b41-19f0-402f-9f23-604c2f61acd7" (UID: "29995b41-19f0-402f-9f23-604c2f61acd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:25.968563 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.968532    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-dshm" pod "29995b41-19f0-402f-9f23-604c2f61acd7" (UID: "29995b41-19f0-402f-9f23-604c2f61acd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:25.968723 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.968697    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29995b41-19f0-402f-9f23-604c2f61acd7-tls-certs" pod "29995b41-19f0-402f-9f23-604c2f61acd7" (UID: "29995b41-19f0-402f-9f23-604c2f61acd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:25.968802 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.968747    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29995b41-19f0-402f-9f23-604c2f61acd7-kube-api-access-m5j8c" pod "29995b41-19f0-402f-9f23-604c2f61acd7" (UID: "29995b41-19f0-402f-9f23-604c2f61acd7"). InnerVolumeSpecName "kube-api-access-m5j8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:25.976924 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.976883    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-tmp-dir" pod "29995b41-19f0-402f-9f23-604c2f61acd7" (UID: "29995b41-19f0-402f-9f23-604c2f61acd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:25.993083 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:25.993051    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-kserve-provision-location" pod "29995b41-19f0-402f-9f23-604c2f61acd7" (UID: "29995b41-19f0-402f-9f23-604c2f61acd7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:26.043868 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.043833    4494 generic.go:356] "Generic (PLEG): container finished" podID="29995b41-19f0-402f-9f23-604c2f61acd7" containerID="0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410" exitCode=0
Oct 30 04:03:26.044073 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.043893    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"
Oct 30 04:03:26.044073 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.043923    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" event={"ID":"29995b41-19f0-402f-9f23-604c2f61acd7","Type":"ContainerDied","Data":"0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410"}
Oct 30 04:03:26.044073 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.043986    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm" event={"ID":"29995b41-19f0-402f-9f23-604c2f61acd7","Type":"ContainerDied","Data":"5052ebc55d5dc3176b5803c020284d5cd9ddaf123d717811a78ae76dd921d743"}
Oct 30 04:03:26.044073 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.044018    4494 scope.go:122] "RemoveContainer" containerID="0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410"
Oct 30 04:03:26.053511 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.053492    4494 scope.go:122] "RemoveContainer" containerID="29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2"
Oct 30 04:03:26.062688 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.062666    4494 scope.go:122] "RemoveContainer" containerID="0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410"
Oct 30 04:03:26.063002 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:03:26.062979    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410\": container with ID starting with 0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410 not found: ID does not exist" containerID="0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410"
Oct 30 04:03:26.063099 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.063016    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410"} err="failed to get container status \"0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410\": rpc error: code = NotFound desc = could not find container \"0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410\": container with ID starting with 0fbf4de2eba1d30ca027030e674d142e4e1c281c1d5567e9ee3bbc4cfc09f410 not found: ID does not exist"
Oct 30 04:03:26.063099 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.063047    4494 scope.go:122] "RemoveContainer" containerID="29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2"
Oct 30 04:03:26.063342 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:03:26.063322    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2\": container with ID starting with 29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2 not found: ID does not exist" containerID="29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2"
Oct 30 04:03:26.063393 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.063348    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2"} err="failed to get container status \"29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2\": rpc error: code = NotFound desc = could not find container \"29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2\": container with ID starting with 29e883ff3f90255866ea7c081d10a3ea7142c9a5969bd9bf9d72a9e221de28f2 not found: ID does not exist"
Oct 30 04:03:26.066778 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.066757    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.066872 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.066780    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.066872 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.066792    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.066872 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.066807    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.066872 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.066821    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5j8c\" (UniqueName: \"kubernetes.io/projected/29995b41-19f0-402f-9f23-604c2f61acd7-kube-api-access-m5j8c\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.066872 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.066834    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29995b41-19f0-402f-9f23-604c2f61acd7-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.066872 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.066857    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29995b41-19f0-402f-9f23-604c2f61acd7-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:26.097360 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.097325    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"]
Oct 30 04:03:26.100415 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:26.100379    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7f99dc589d-vj4dm"]
Oct 30 04:03:27.927144 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:27.927104    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" path="/var/lib/kubelet/pods/29995b41-19f0-402f-9f23-604c2f61acd7/volumes"
Oct 30 04:03:34.341315 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341279    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"]
Oct 30 04:03:34.341732 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341637    4494 cpu_manager.go:414] "containerMap: removing container" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main"
Oct 30 04:03:34.341732 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341654    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main"
Oct 30 04:03:34.341732 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341664    4494 cpu_manager.go:414] "containerMap: removing container" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="storage-initializer"
Oct 30 04:03:34.341732 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341669    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="storage-initializer"
Oct 30 04:03:34.341732 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341678    4494 cpu_manager.go:414] "containerMap: removing container" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="storage-initializer"
Oct 30 04:03:34.341732 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341683    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="storage-initializer"
Oct 30 04:03:34.341732 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341698    4494 cpu_manager.go:414] "containerMap: removing container" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main"
Oct 30 04:03:34.341732 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341703    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main"
Oct 30 04:03:34.342009 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341769    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d56520-7b23-4b1f-ad90-242116bc3de0" containerName="main"
Oct 30 04:03:34.342009 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.341781    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="29995b41-19f0-402f-9f23-604c2f61acd7" containerName="main"
Oct 30 04:03:34.346494 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.346474    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:34.362624 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.362587    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 30 04:03:34.374913 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.374863    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Oct 30 04:03:34.374913 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.374864    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-kb9nj\""
Oct 30 04:03:34.375133 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.374868    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Oct 30 04:03:34.376459 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.376434    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"]
Oct 30 04:03:34.441702 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.441661    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrh74\" (UniqueName: \"kubernetes.io/projected/2727cba5-b1e4-40a5-9da8-2ccd7b15766f-kube-api-access-vrh74\") pod \"e2e-pvc-lora-download-mmzpf\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:34.542745 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.542701    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrh74\" (UniqueName: \"kubernetes.io/projected/2727cba5-b1e4-40a5-9da8-2ccd7b15766f-kube-api-access-vrh74\") pod \"e2e-pvc-lora-download-mmzpf\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:34.566716 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:34.566685    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrh74\" (UniqueName: \"kubernetes.io/projected/2727cba5-b1e4-40a5-9da8-2ccd7b15766f-kube-api-access-vrh74\") pod \"e2e-pvc-lora-download-mmzpf\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:41.301498 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:41.301455    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"e2e-pvc-lora-download-mmzpf\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:41.331364 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:41.331326    4494 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"e2e-pvc-lora-download-mmzpf\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:41.402850 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:41.402807    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"e2e-pvc-lora-download-mmzpf\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:41.403105 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:41.402987    4494 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"e2e-pvc-lora-download-mmzpf\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:41.417528 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:41.417492    4494 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"e2e-pvc-lora-download-mmzpf\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") DevicePath \"csi-76e6bc02cca9b8b1ecadb59f15442f9b6f99a06aced7b98a3d5e109ebe4b34f1\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:41.653875 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:41.653838    4494 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"e2e-pvc-lora-download-mmzpf\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4d71484972cf8aa01687f420c1228dde81324c7640fbd2b1f17c91ad1c9d0716/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:41.678946 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:41.678917    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"e2e-pvc-lora-download-mmzpf\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:41.871730 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:41.871693    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-kb9nj\""
Oct 30 04:03:41.877925 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:41.877870    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:42.177004 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:42.176945    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"]
Oct 30 04:03:42.177864 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:03:42.177836    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2727cba5_b1e4_40a5_9da8_2ccd7b15766f.slice/crio-8cd134c29a7322379ebf759fcae5827da69e39ea90f7c0c88065dd1f756209bc WatchSource:0}: Error finding container 8cd134c29a7322379ebf759fcae5827da69e39ea90f7c0c88065dd1f756209bc: Status 404 returned error can't find the container with id 8cd134c29a7322379ebf759fcae5827da69e39ea90f7c0c88065dd1f756209bc
Oct 30 04:03:42.193980 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:42.193945    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 30 04:03:43.124177 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:43.124128    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf" event={"ID":"2727cba5-b1e4-40a5-9da8-2ccd7b15766f","Type":"ContainerStarted","Data":"838c7cc9c2efe281be6cb253fa13e831d477f1fc4b5d11c9f5793e10198cc001"}
Oct 30 04:03:43.124729 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:43.124190    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf" event={"ID":"2727cba5-b1e4-40a5-9da8-2ccd7b15766f","Type":"ContainerStarted","Data":"8cd134c29a7322379ebf759fcae5827da69e39ea90f7c0c88065dd1f756209bc"}
Oct 30 04:03:43.166422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:43.166345    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf" podStartSLOduration=12.166328457 podStartE2EDuration="12.166328457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:43.165349196 +0000 UTC m=+4087.371529419" watchObservedRunningTime="2026-10-30 04:03:43.166328457 +0000 UTC m=+4087.372508676"
Oct 30 04:03:45.133388 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:45.133351    4494 generic.go:356] "Generic (PLEG): container finished" podID="2727cba5-b1e4-40a5-9da8-2ccd7b15766f" containerID="838c7cc9c2efe281be6cb253fa13e831d477f1fc4b5d11c9f5793e10198cc001" exitCode=0
Oct 30 04:03:45.133388 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:45.133392    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf" event={"ID":"2727cba5-b1e4-40a5-9da8-2ccd7b15766f","Type":"ContainerDied","Data":"838c7cc9c2efe281be6cb253fa13e831d477f1fc4b5d11c9f5793e10198cc001"}
Oct 30 04:03:46.266778 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:46.266748    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:46.448258 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:46.448170    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2727cba5-b1e4-40a5-9da8-2ccd7b15766f-kube-api-access-vrh74\" (UniqueName: \"kubernetes.io/projected/2727cba5-b1e4-40a5-9da8-2ccd7b15766f-kube-api-access-vrh74\") pod \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") "
Oct 30 04:03:46.448415 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:46.448326    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\" (UID: \"2727cba5-b1e4-40a5-9da8-2ccd7b15766f\") "
Oct 30 04:03:46.450630 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:46.450601    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2727cba5-b1e4-40a5-9da8-2ccd7b15766f-kube-api-access-vrh74" pod "2727cba5-b1e4-40a5-9da8-2ccd7b15766f" (UID: "2727cba5-b1e4-40a5-9da8-2ccd7b15766f"). InnerVolumeSpecName "kube-api-access-vrh74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:46.452327 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:46.452304    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7" pod "2727cba5-b1e4-40a5-9da8-2ccd7b15766f" (UID: "2727cba5-b1e4-40a5-9da8-2ccd7b15766f"). InnerVolumeSpecName "pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:03:46.549193 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:46.549151    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrh74\" (UniqueName: \"kubernetes.io/projected/2727cba5-b1e4-40a5-9da8-2ccd7b15766f-kube-api-access-vrh74\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:03:46.549406 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:46.549209    4494 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") on node \"ip-10-0-130-130.ec2.internal\" "
Oct 30 04:03:47.142774 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:47.142723    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf" event={"ID":"2727cba5-b1e4-40a5-9da8-2ccd7b15766f","Type":"ContainerDied","Data":"8cd134c29a7322379ebf759fcae5827da69e39ea90f7c0c88065dd1f756209bc"}
Oct 30 04:03:47.142774 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:47.142774    4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cd134c29a7322379ebf759fcae5827da69e39ea90f7c0c88065dd1f756209bc"
Oct 30 04:03:47.142774 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:47.142748    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"
Oct 30 04:03:47.563828 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:47.563808    4494 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7") on node "ip-10-0-130-130.ec2.internal"
Oct 30 04:03:47.658970 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:47.658933    4494 reconciler_common.go:299] "Volume detached for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"csi-76e6bc02cca9b8b1ecadb59f15442f9b6f99a06aced7b98a3d5e109ebe4b34f1\""
Oct 30 04:03:49.293698 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.293656    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"]
Oct 30 04:03:49.294293 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.294255    4494 cpu_manager.go:414] "containerMap: removing container" podUID="2727cba5-b1e4-40a5-9da8-2ccd7b15766f" containerName="storage-initializer"
Oct 30 04:03:49.294293 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.294282    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2727cba5-b1e4-40a5-9da8-2ccd7b15766f" containerName="storage-initializer"
Oct 30 04:03:49.294445 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.294389    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="2727cba5-b1e4-40a5-9da8-2ccd7b15766f" containerName="storage-initializer"
Oct 30 04:03:49.298026 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.297992    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.313167 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.313135    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Oct 30 04:03:49.313167 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.313165    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Oct 30 04:03:49.313383 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.313165    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Oct 30 04:03:49.313383 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.313183    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-48r92\""
Oct 30 04:03:49.330019 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.329986    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"]
Oct 30 04:03:49.371626 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.371577    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-token\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.371626 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.371623    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-envoy\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.371846 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.371648    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-data\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.371846 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.371698    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.371846 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.371826    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-socket\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.372021 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.371868    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.372021 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.371941    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-certs\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.372021 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.372007    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-credential-socket\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.372135 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.372042    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cq6n\" (UniqueName: \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-kube-api-access-8cq6n\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473281 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473218    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-data\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473281 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473282    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473559 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473328    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-socket\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473559 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473368    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473559 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473401    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-certs\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473559 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473435    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-credential-socket\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473559 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473459    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cq6n\" (UniqueName: \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-kube-api-access-8cq6n\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473559 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473517    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-token\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473559 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473546    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-envoy\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473822 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473700    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-data\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.473822 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.473760    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-socket\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.474081 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.474060    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-certs\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.474189 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.474166    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-credential-socket\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.474416 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.474210    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.476215 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.476192    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.476346 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.476326    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-envoy\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.498771 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.498735    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-token\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.498947 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.498798    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cq6n\" (UniqueName: \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-kube-api-access-8cq6n\") pod \"router-gateway-2-openshift-default-7bd678d5fc-xk4x5\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.615048 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.614891    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:49.927730 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:03:49.927702    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54801ec4_fab8_4c3e_befb_2b0b16d74f9e.slice/crio-adc031fa7edfc8fd5085d19748b995040ae11c96c71c8771044171cad1fd6149 WatchSource:0}: Error finding container adc031fa7edfc8fd5085d19748b995040ae11c96c71c8771044171cad1fd6149: Status 404 returned error can't find the container with id adc031fa7edfc8fd5085d19748b995040ae11c96c71c8771044171cad1fd6149
Oct 30 04:03:49.929772 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.929748    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"]
Oct 30 04:03:49.929772 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.929750    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:03:49.929923 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.929823    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:03:49.929923 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:49.929857    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:03:50.158254 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:50.158204    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" event={"ID":"54801ec4-fab8-4c3e-befb-2b0b16d74f9e","Type":"ContainerStarted","Data":"52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a"}
Oct 30 04:03:50.158448 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:50.158275    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" event={"ID":"54801ec4-fab8-4c3e-befb-2b0b16d74f9e","Type":"ContainerStarted","Data":"adc031fa7edfc8fd5085d19748b995040ae11c96c71c8771044171cad1fd6149"}
Oct 30 04:03:50.210597 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:50.210488    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" podStartSLOduration=1.210473337 podStartE2EDuration="1.210473337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:50.20929965 +0000 UTC m=+4094.415479870" watchObservedRunningTime="2026-10-30 04:03:50.210473337 +0000 UTC m=+4094.416653555"
Oct 30 04:03:50.615683 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:50.615637    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:50.620500 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:50.620475    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:51.163230 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:51.163195    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:51.164206 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:03:51.164186    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:03:54.105444 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:03:54.105399    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:04:01.231413 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:01.231378    4494 generic.go:356] "Generic (PLEG): container finished" podID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerID="52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a" exitCode=0
Oct 30 04:04:01.231869 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:01.231446    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" event={"ID":"54801ec4-fab8-4c3e-befb-2b0b16d74f9e","Type":"ContainerDied","Data":"52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a"}
Oct 30 04:04:01.231869 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:01.231810    4494 scope.go:122] "RemoveContainer" containerID="52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a"
Oct 30 04:04:01.232853 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:01.232825    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:04:01.232955 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:01.232939    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:04:01.232998 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:01.232975    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:04:02.245130 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:02.245090    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" event={"ID":"54801ec4-fab8-4c3e-befb-2b0b16d74f9e","Type":"ContainerStarted","Data":"04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07"}
Oct 30 04:04:02.615324 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:02.615277    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:04:02.616570 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:02.616528    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 04:04:03.616232 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:03.616187    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 04:04:03.900992 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:03.897242    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"]
Oct 30 04:04:03.909251 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:03.909224    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:03.930029 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:03.929995    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"]
Oct 30 04:04:03.933997 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:03.933951    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Oct 30 04:04:03.933997 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:03.933989    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:04:03.933997 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:03.933989    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Oct 30 04:04:03.934252 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:03.934047    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-kb9nj\""
Oct 30 04:04:03.934252 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:03.934215    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Oct 30 04:04:03.934317 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:03.934269    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Oct 30 04:04:04.013553 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.013509    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-dshm\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.013553 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.013566    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.013809 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.013644    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.013809 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.013690    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.013954 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.013819    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-home\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.014020 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.013946    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.014020 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.013992    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.114442 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.114400    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.114672 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.114480    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-home\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.114672 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.114511    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.114672 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.114535    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.114672 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.114553    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-dshm\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.114672 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.114575    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8s7r\" (UniqueName: \"kubernetes.io/projected/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kube-api-access-b8s7r\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.114672 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.114593    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.114672 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.114609    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.115118 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.114947    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-home\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.115302 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.115258    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.115302 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.115287    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.115302 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.115296    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.115631 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.115606    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.117309 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.117289    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-dshm\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.117494 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.117477    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.215751 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.215645    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8s7r\" (UniqueName: \"kubernetes.io/projected/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kube-api-access-b8s7r\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.242366 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.242339    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8s7r\" (UniqueName: \"kubernetes.io/projected/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kube-api-access-b8s7r\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:04.616286 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:04.616236    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 04:04:05.582929 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.579508    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"]
Oct 30 04:04:05.597229 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.596323    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.612149 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.610860    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"]
Oct 30 04:04:05.616281 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.616152    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 04:04:05.626994 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.626961    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:04:05.627520 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.627057    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Oct 30 04:04:05.627520 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.627191    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Oct 30 04:04:05.627520 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.627262    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-jmw65\""
Oct 30 04:04:05.729947 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.729835    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7465f30-acd0-4796-bfa6-03d7d2793db4-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.729947 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.729877    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7465f30-acd0-4796-bfa6-03d7d2793db4-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.729947 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.729913    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.730177 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.729971    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.730177 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.730028    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.730177 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.730072    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.730177 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.730093    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-home\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.730307 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.730178    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqt6\" (UniqueName: \"kubernetes.io/projected/c7465f30-acd0-4796-bfa6-03d7d2793db4-kube-api-access-4rqt6\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.831163 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.831120    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.831342 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.831195    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.831342 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.831217    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-home\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.831342 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.831247    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqt6\" (UniqueName: \"kubernetes.io/projected/c7465f30-acd0-4796-bfa6-03d7d2793db4-kube-api-access-4rqt6\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.831342 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.831282    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7465f30-acd0-4796-bfa6-03d7d2793db4-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.831342 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.831304    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7465f30-acd0-4796-bfa6-03d7d2793db4-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.831342 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.831324    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.831342 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.831341    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.831719 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.831687    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.832112 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.832081    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-home\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.832112 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.832102    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.832258 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.832164    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.832362 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.832341    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7465f30-acd0-4796-bfa6-03d7d2793db4-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.834286 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.834267    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7465f30-acd0-4796-bfa6-03d7d2793db4-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.834454 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.834430    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.855815 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.855771    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqt6\" (UniqueName: \"kubernetes.io/projected/c7465f30-acd0-4796-bfa6-03d7d2793db4-kube-api-access-4rqt6\") pod \"router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:05.923949 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:05.923887    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:06.164112 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:06.164069    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:04:06.262091 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:06.262060    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"]
Oct 30 04:04:06.262330 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:04:06.262306    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7465f30_acd0_4796_bfa6_03d7d2793db4.slice/crio-fdcf029afb3d9c1b911d0ec742127ab7a10510dc91650834dfd8249200252694 WatchSource:0}: Error finding container fdcf029afb3d9c1b911d0ec742127ab7a10510dc91650834dfd8249200252694: Status 404 returned error can't find the container with id fdcf029afb3d9c1b911d0ec742127ab7a10510dc91650834dfd8249200252694
Oct 30 04:04:06.616272 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:06.616226    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 04:04:07.270601 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:07.270555    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" event={"ID":"c7465f30-acd0-4796-bfa6-03d7d2793db4","Type":"ContainerStarted","Data":"81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51"}
Oct 30 04:04:07.271051 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:07.270673    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" event={"ID":"c7465f30-acd0-4796-bfa6-03d7d2793db4","Type":"ContainerStarted","Data":"fdcf029afb3d9c1b911d0ec742127ab7a10510dc91650834dfd8249200252694"}
Oct 30 04:04:07.619360 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:07.619321    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:04:07.620314 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:07.620289    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:04:09.281982 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:09.281952    4494 generic.go:356] "Generic (PLEG): container finished" podID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerID="81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51" exitCode=0
Oct 30 04:04:09.282479 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:09.282029    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" event={"ID":"c7465f30-acd0-4796-bfa6-03d7d2793db4","Type":"ContainerDied","Data":"81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51"}
Oct 30 04:04:10.289492 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:10.289458    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" event={"ID":"c7465f30-acd0-4796-bfa6-03d7d2793db4","Type":"ContainerStarted","Data":"657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0"}
Oct 30 04:04:10.361955 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:10.361874    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podStartSLOduration=2.874496649 podStartE2EDuration="5.361858003s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.487361354s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:10.349400301 +0000 UTC m=+4114.555580534" watchObservedRunningTime="2026-10-30 04:04:10.361858003 +0000 UTC m=+4114.568038221"
Oct 30 04:04:11.988190 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:11.988145    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:12.017807 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:12.017773    4494 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:12.089105 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:12.089058    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:12.089311 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:12.089216    4494 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:12.109007 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:12.108968    4494 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") DevicePath \"csi-76e6bc02cca9b8b1ecadb59f15442f9b6f99a06aced7b98a3d5e109ebe4b34f1\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:12.212582 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:12.212547    4494 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4d71484972cf8aa01687f420c1228dde81324c7640fbd2b1f17c91ad1c9d0716/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:12.243545 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:12.243462    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:12.327121 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:12.327078    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:12.650460 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:12.650422    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"]
Oct 30 04:04:12.651461 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:04:12.651427    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e98545_6d62_4f36_8cd7_cdec6013cf7a.slice/crio-1e3f3010b1a4e03a6e14a6f9632d50b86bab56b2b8fbdb85f2509a594fc05641 WatchSource:0}: Error finding container 1e3f3010b1a4e03a6e14a6f9632d50b86bab56b2b8fbdb85f2509a594fc05641: Status 404 returned error can't find the container with id 1e3f3010b1a4e03a6e14a6f9632d50b86bab56b2b8fbdb85f2509a594fc05641
Oct 30 04:04:13.305353 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:13.305311    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" event={"ID":"d8e98545-6d62-4f36-8cd7-cdec6013cf7a","Type":"ContainerStarted","Data":"c482d198cdb266fc8e06fda34b3294c6e5ccfc960c11f192c73fb92582170665"}
Oct 30 04:04:13.305754 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:13.305379    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" event={"ID":"d8e98545-6d62-4f36-8cd7-cdec6013cf7a","Type":"ContainerStarted","Data":"1e3f3010b1a4e03a6e14a6f9632d50b86bab56b2b8fbdb85f2509a594fc05641"}
Oct 30 04:04:15.315633 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:15.315598    4494 generic.go:356] "Generic (PLEG): container finished" podID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerID="c482d198cdb266fc8e06fda34b3294c6e5ccfc960c11f192c73fb92582170665" exitCode=0
Oct 30 04:04:15.316062 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:15.315684    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" event={"ID":"d8e98545-6d62-4f36-8cd7-cdec6013cf7a","Type":"ContainerDied","Data":"c482d198cdb266fc8e06fda34b3294c6e5ccfc960c11f192c73fb92582170665"}
Oct 30 04:04:15.925956 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:15.925890    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:04:15.928880 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:15.928846    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:15.929087 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:15.928937    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:04:16.322580 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:16.322538    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" event={"ID":"d8e98545-6d62-4f36-8cd7-cdec6013cf7a","Type":"ContainerStarted","Data":"46e2fbec5d093d86f2b4fac4cbda1d5b6bb5fe51f10f09f4ea2641fe6d3f2d5e"}
Oct 30 04:04:16.328059 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:16.328035    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:16.396790 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:16.396716    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" podStartSLOduration=11.049732613 podStartE2EDuration="13.396695625s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.346963012s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:16.383208035 +0000 UTC m=+4120.589388251" watchObservedRunningTime="2026-10-30 04:04:16.396695625 +0000 UTC m=+4120.602875843"
Oct 30 04:04:22.327951 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:22.327912    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:04:22.329754 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:22.329716    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Oct 30 04:04:24.085403 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:04:24.085362    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:04:25.924359 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:25.924312    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:04:32.328011 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:32.327954    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Oct 30 04:04:35.924370 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:35.924319    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:04:42.327593 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:42.327542    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Oct 30 04:04:45.924443 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:45.924386    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:04:51.564740 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.564416    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"]
Oct 30 04:04:51.568461 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.568431    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.591691 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.591636    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f46c43-2286-48da-bb3c-acf0bd23720a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.591931 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.591698    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.591931 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.591872    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.592081 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.591988    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.592143 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.592089    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.592143 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.592133    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.592255 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.592159    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlw5\" (UniqueName: \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-kube-api-access-htlw5\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.592313 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.592262    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.592313 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.592293    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.598400 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.598355    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"]
Oct 30 04:04:51.670695 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.670631    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 04:04:51.670695 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.670656    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 04:04:51.670999 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.670661    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 04:04:51.670999 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.670751    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-pzw79\""
Oct 30 04:04:51.693321 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.693281    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.693321 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.693322    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlw5\" (UniqueName: \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-kube-api-access-htlw5\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.693498 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.693375    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.693498 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.693409    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.693498 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.693442    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f46c43-2286-48da-bb3c-acf0bd23720a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.693498 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.693469    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.693711 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.693511    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.693711 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.693554    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.693711 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.693616    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.694123 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.694094    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.694354 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.694291    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.694480 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.694391    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.694589 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.694566    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f46c43-2286-48da-bb3c-acf0bd23720a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.694918 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.694874    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.696328 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.696308    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.696487 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.696466    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.717518 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.717481    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.717826 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.717796    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlw5\" (UniqueName: \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-kube-api-access-htlw5\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:51.887112 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:51.886981    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:52.207664 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:52.207626    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"]
Oct 30 04:04:52.207820 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:04:52.207789    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f46c43_2286_48da_bb3c_acf0bd23720a.slice/crio-096efc6b9b3865ff892a066026b431274e6e936a99629af67daef5d94cf5e9ec WatchSource:0}: Error finding container 096efc6b9b3865ff892a066026b431274e6e936a99629af67daef5d94cf5e9ec: Status 404 returned error can't find the container with id 096efc6b9b3865ff892a066026b431274e6e936a99629af67daef5d94cf5e9ec
Oct 30 04:04:52.210417 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:52.210381    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:04:52.210535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:52.210467    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:04:52.210535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:52.210516    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:04:52.328291 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:52.328245    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Oct 30 04:04:52.520620 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:52.520509    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" event={"ID":"33f46c43-2286-48da-bb3c-acf0bd23720a","Type":"ContainerStarted","Data":"9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39"}
Oct 30 04:04:52.520620 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:52.520594    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" event={"ID":"33f46c43-2286-48da-bb3c-acf0bd23720a","Type":"ContainerStarted","Data":"096efc6b9b3865ff892a066026b431274e6e936a99629af67daef5d94cf5e9ec"}
Oct 30 04:04:52.570054 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:52.569972    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" podStartSLOduration=1.569951458 podStartE2EDuration="1.569951458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:52.565929837 +0000 UTC m=+4156.772110056" watchObservedRunningTime="2026-10-30 04:04:52.569951458 +0000 UTC m=+4156.776131680"
Oct 30 04:04:52.891079 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:52.891016    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:53.896094 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:53.896044    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 04:04:53.896792 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:53.896752    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 04:04:54.111931 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:04:54.108181    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:04:54.888938 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:54.888333    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 04:04:54.888938 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:54.888400    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 04:04:55.888422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:55.888374    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 04:04:55.889121 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:55.889081    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 04:04:55.924455 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:55.924413    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:04:56.891924 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:56.891359    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 04:04:56.891924 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:56.891429    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 04:04:57.891929 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:57.891375    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Oct 30 04:04:57.891929 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:57.891450    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Oct 30 04:04:58.892201 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:58.892159    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:58.892755 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:58.892728    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:04:58.893744 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:04:58.893718    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:05:02.328088 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:02.328033    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Oct 30 04:05:05.924712 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:05.924670    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:05:06.601731 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.601678    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"]
Oct 30 04:05:06.606326 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.606297    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.635931 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.635871    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"]
Oct 30 04:05:06.656594 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.656552    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.656594 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.656597    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b08259-c22f-4be6-a144-4049edf81dca-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.656886 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.656617    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b08259-c22f-4be6-a144-4049edf81dca-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.656886 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.656641    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.656886 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.656816    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.657098 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.656890    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.657098 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.656962    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbslj\" (UniqueName: \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-kube-api-access-hbslj\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.657098 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.657019    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.657098 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.657040    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.758713 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.758658    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.758713 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.758724    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b08259-c22f-4be6-a144-4049edf81dca-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759008 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.758750    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b08259-c22f-4be6-a144-4049edf81dca-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759008 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.758781    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759008 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.758811    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759008 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.758871    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759008 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.758981    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbslj\" (UniqueName: \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-kube-api-access-hbslj\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759289 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.759063    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759289 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.759093    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759289 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.759211    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759530 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.759503    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759688 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.759661    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.759958 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.759935    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.760072 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.760048    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b08259-c22f-4be6-a144-4049edf81dca-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.762297 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.762263    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.762433 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.762373    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b08259-c22f-4be6-a144-4049edf81dca-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.783441 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.783394    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.783441 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.783416    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbslj\" (UniqueName: \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-kube-api-access-hbslj\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:06.922433 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:06.922315    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:07.244275 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:05:07.244230    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b08259_c22f_4be6_a144_4049edf81dca.slice/crio-4eee5d6421844a76c24f57e06993a7fc5df71173cd8e471bf0b55bb8425699ac WatchSource:0}: Error finding container 4eee5d6421844a76c24f57e06993a7fc5df71173cd8e471bf0b55bb8425699ac: Status 404 returned error can't find the container with id 4eee5d6421844a76c24f57e06993a7fc5df71173cd8e471bf0b55bb8425699ac
Oct 30 04:05:07.244676 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:07.244504    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"]
Oct 30 04:05:07.246832 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:07.246790    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:05:07.246962 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:07.246890    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:05:07.247027 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:07.246971    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:05:07.604466 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:07.604418    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" event={"ID":"c4b08259-c22f-4be6-a144-4049edf81dca","Type":"ContainerStarted","Data":"4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45"}
Oct 30 04:05:07.604700 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:07.604498    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" event={"ID":"c4b08259-c22f-4be6-a144-4049edf81dca","Type":"ContainerStarted","Data":"4eee5d6421844a76c24f57e06993a7fc5df71173cd8e471bf0b55bb8425699ac"}
Oct 30 04:05:07.674571 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:07.674510    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" podStartSLOduration=1.674493717 podStartE2EDuration="1.674493717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:07.661854937 +0000 UTC m=+4171.868035180" watchObservedRunningTime="2026-10-30 04:05:07.674493717 +0000 UTC m=+4171.880674063"
Oct 30 04:05:07.927566 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:07.927483    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:08.924353 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:08.924304    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 04:05:08.924807 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:08.924375    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 04:05:09.923717 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:09.923667    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 04:05:09.923939 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:09.923737    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 04:05:10.923685 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:10.923641    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 04:05:10.924166 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:10.923706    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 04:05:11.923577 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:11.923538    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 04:05:11.923774 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:11.923605    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 04:05:12.328689 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:12.328630    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Oct 30 04:05:12.923497 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:12.923460    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 04:05:12.923689 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:12.923521    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 04:05:13.929701 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:13.929669    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:13.930268 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:13.930048    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:13.931300 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:13.931278    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:05:15.924676 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:15.924636    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:05:22.327874 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:22.327823    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Oct 30 04:05:24.087632 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:05:24.087576    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:05:25.924505 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:25.924460    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:05:32.328266 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:32.328218    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": dial tcp 10.134.0.82:8000: connect: connection refused"
Oct 30 04:05:35.924704 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:35.924644    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:05:42.337851 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:42.337812    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:05:42.346242 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:42.346208    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:05:45.925667 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:45.925622    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:05:54.083762 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:05:54.083708    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:05:55.924332 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:05:55.924287    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:06:05.925093 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:05.925046    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:06:15.924590 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:15.924545    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:06:24.083603 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:06:24.083552    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:06:25.924635 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:25.924599    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:06:35.924406 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:35.924343    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": dial tcp 10.134.0.83:8000: connect: connection refused"
Oct 30 04:06:36.056846 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:36.056806    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr_d8e98545-6d62-4f36-8cd7-cdec6013cf7a/storage-initializer/0.log"
Oct 30 04:06:36.098433 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:36.098400    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr_d8e98545-6d62-4f36-8cd7-cdec6013cf7a/main/0.log"
Oct 30 04:06:39.573461 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:39.573420    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"]
Oct 30 04:06:45.935079 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:45.935047    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:06:45.943035 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:45.943006    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:06:54.087478 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:06:54.087370    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:06:54.598447 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:54.598392    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main" containerID="cri-o://46e2fbec5d093d86f2b4fac4cbda1d5b6bb5fe51f10f09f4ea2641fe6d3f2d5e" gracePeriod=45
Oct 30 04:06:55.150576 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.150540    4494 generic.go:356] "Generic (PLEG): container finished" podID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerID="46e2fbec5d093d86f2b4fac4cbda1d5b6bb5fe51f10f09f4ea2641fe6d3f2d5e" exitCode=0
Oct 30 04:06:55.150984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.150608    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" event={"ID":"d8e98545-6d62-4f36-8cd7-cdec6013cf7a","Type":"ContainerDied","Data":"46e2fbec5d093d86f2b4fac4cbda1d5b6bb5fe51f10f09f4ea2641fe6d3f2d5e"}
Oct 30 04:06:55.268932 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.268884    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:06:55.378414 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.378325    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") pod \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") "
Oct 30 04:06:55.378414 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.378370    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-model-cache\") pod \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") "
Oct 30 04:06:55.378414 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.378400    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tls-certs\") pod \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") "
Oct 30 04:06:55.378733 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.378456    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tmp-dir\") pod \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") "
Oct 30 04:06:55.378733 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.378479    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kserve-provision-location\") pod \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") "
Oct 30 04:06:55.378733 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.378526    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-dshm\") pod \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") "
Oct 30 04:06:55.378733 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.378568    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kube-api-access-b8s7r\" (UniqueName: \"kubernetes.io/projected/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kube-api-access-b8s7r\") pod \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") "
Oct 30 04:06:55.378733 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.378608    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-home\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-home\") pod \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") "
Oct 30 04:06:55.378733 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.378641    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-cabundle-cert\") pod \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\" (UID: \"d8e98545-6d62-4f36-8cd7-cdec6013cf7a\") "
Oct 30 04:06:55.379025 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.378801    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-model-cache" pod "d8e98545-6d62-4f36-8cd7-cdec6013cf7a" (UID: "d8e98545-6d62-4f36-8cd7-cdec6013cf7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:55.379081 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.379023    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:06:55.379158 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.379134    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-cabundle-cert" pod "d8e98545-6d62-4f36-8cd7-cdec6013cf7a" (UID: "d8e98545-6d62-4f36-8cd7-cdec6013cf7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:06:55.379344 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.379324    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-home" pod "d8e98545-6d62-4f36-8cd7-cdec6013cf7a" (UID: "d8e98545-6d62-4f36-8cd7-cdec6013cf7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:55.381279 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.381250    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tls-certs" pod "d8e98545-6d62-4f36-8cd7-cdec6013cf7a" (UID: "d8e98545-6d62-4f36-8cd7-cdec6013cf7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:06:55.382171 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.382141    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-dshm" pod "d8e98545-6d62-4f36-8cd7-cdec6013cf7a" (UID: "d8e98545-6d62-4f36-8cd7-cdec6013cf7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:55.382330 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.382259    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kube-api-access-b8s7r" pod "d8e98545-6d62-4f36-8cd7-cdec6013cf7a" (UID: "d8e98545-6d62-4f36-8cd7-cdec6013cf7a"). InnerVolumeSpecName "kube-api-access-b8s7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:06:55.389348 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.389321    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7" pod "d8e98545-6d62-4f36-8cd7-cdec6013cf7a" (UID: "d8e98545-6d62-4f36-8cd7-cdec6013cf7a"). InnerVolumeSpecName "pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:06:55.399375 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.399331    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tmp-dir" pod "d8e98545-6d62-4f36-8cd7-cdec6013cf7a" (UID: "d8e98545-6d62-4f36-8cd7-cdec6013cf7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:55.400877 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.400848    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kserve-provision-location" pod "d8e98545-6d62-4f36-8cd7-cdec6013cf7a" (UID: "d8e98545-6d62-4f36-8cd7-cdec6013cf7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:06:55.479821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.479772    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:06:55.479821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.479814    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:06:55.479821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.479824    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:06:55.479821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.479835    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8s7r\" (UniqueName: \"kubernetes.io/projected/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-kube-api-access-b8s7r\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:06:55.480236 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.479844    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:06:55.480236 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.479852    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:06:55.480236 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.479885    4494 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") on node \"ip-10-0-130-130.ec2.internal\" "
Oct 30 04:06:55.480236 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.479934    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e98545-6d62-4f36-8cd7-cdec6013cf7a-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:06:55.490621 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.490596    4494 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7") on node "ip-10-0-130-130.ec2.internal"
Oct 30 04:06:55.581327 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:55.581291    4494 reconciler_common.go:299] "Volume detached for volume \"pvc-d5028d53-bbef-47a5-b6bd-b72c86b9672a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c3feb231763802f7\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"csi-76e6bc02cca9b8b1ecadb59f15442f9b6f99a06aced7b98a3d5e109ebe4b34f1\""
Oct 30 04:06:56.158451 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:56.158421    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"
Oct 30 04:06:56.158997 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:56.158421    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr" event={"ID":"d8e98545-6d62-4f36-8cd7-cdec6013cf7a","Type":"ContainerDied","Data":"1e3f3010b1a4e03a6e14a6f9632d50b86bab56b2b8fbdb85f2509a594fc05641"}
Oct 30 04:06:56.158997 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:56.158569    4494 scope.go:122] "RemoveContainer" containerID="46e2fbec5d093d86f2b4fac4cbda1d5b6bb5fe51f10f09f4ea2641fe6d3f2d5e"
Oct 30 04:06:56.168082 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:56.168053    4494 scope.go:122] "RemoveContainer" containerID="c482d198cdb266fc8e06fda34b3294c6e5ccfc960c11f192c73fb92582170665"
Oct 30 04:06:56.228406 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:56.228296    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"]
Oct 30 04:06:56.241230 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:56.241185    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-5945f8f45-7hhsr"]
Oct 30 04:06:56.420233 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:56.420200    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"]
Oct 30 04:06:57.927441 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:06:57.927405    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" path="/var/lib/kubelet/pods/d8e98545-6d62-4f36-8cd7-cdec6013cf7a/volumes"
Oct 30 04:07:03.610306 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.610260    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"]
Oct 30 04:07:03.610751 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.610671    4494 cpu_manager.go:414] "containerMap: removing container" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="storage-initializer"
Oct 30 04:07:03.610751 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.610685    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="storage-initializer"
Oct 30 04:07:03.610751 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.610705    4494 cpu_manager.go:414] "containerMap: removing container" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main"
Oct 30 04:07:03.610751 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.610711    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main"
Oct 30 04:07:03.610966 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.610777    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e98545-6d62-4f36-8cd7-cdec6013cf7a" containerName="main"
Oct 30 04:07:03.613740 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.613715    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:03.628816 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.628784    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Oct 30 04:07:03.628816 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.628782    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Oct 30 04:07:03.629060 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.628782    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Oct 30 04:07:03.641294 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.641249    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-m7n6b\""
Oct 30 04:07:03.642946 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.642912    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"]
Oct 30 04:07:03.758700 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.758659    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnrp\" (UniqueName: \"kubernetes.io/projected/0132b0fc-19ea-4401-a7ca-88c3c0728826-kube-api-access-vhnrp\") pod \"e2e-pvc-lora-download-pjtw2\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:03.859957 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.859892    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnrp\" (UniqueName: \"kubernetes.io/projected/0132b0fc-19ea-4401-a7ca-88c3c0728826-kube-api-access-vhnrp\") pod \"e2e-pvc-lora-download-pjtw2\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:03.884186 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:03.884101    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnrp\" (UniqueName: \"kubernetes.io/projected/0132b0fc-19ea-4401-a7ca-88c3c0728826-kube-api-access-vhnrp\") pod \"e2e-pvc-lora-download-pjtw2\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:04.895089 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:04.895047    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"]
Oct 30 04:07:04.898867 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:04.898832    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-mmzpf"]
Oct 30 04:07:05.926878 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:05.926840    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2727cba5-b1e4-40a5-9da8-2ccd7b15766f" path="/var/lib/kubelet/pods/2727cba5-b1e4-40a5-9da8-2ccd7b15766f/volumes"
Oct 30 04:07:06.082568 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:06.082517    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"e2e-pvc-lora-download-pjtw2\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:06.111421 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:06.111381    4494 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"e2e-pvc-lora-download-pjtw2\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:06.183527 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:06.183425    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"e2e-pvc-lora-download-pjtw2\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:06.183706 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:06.183572    4494 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"e2e-pvc-lora-download-pjtw2\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:06.198339 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:06.198305    4494 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"e2e-pvc-lora-download-pjtw2\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") DevicePath \"csi-db7d3a71899d8761428b694623f5564828c101a903a0d4db5c2f7309332abd93\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:06.422596 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:06.422548    4494 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"e2e-pvc-lora-download-pjtw2\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b52da1cb97f47b80b1e26f8ba092e9c96c79fbcd1368ddcf17139a7a90451da1/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:06.448487 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:06.448397    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"e2e-pvc-lora-download-pjtw2\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:06.625446 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:06.625398    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:06.922527 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:06.922486    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"]
Oct 30 04:07:06.922854 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:07:06.922828    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0132b0fc_19ea_4401_a7ca_88c3c0728826.slice/crio-d6f855d791ce131937389313bfd572bddb2c0b03cdbf65d0ce73b8481ddce7a3 WatchSource:0}: Error finding container d6f855d791ce131937389313bfd572bddb2c0b03cdbf65d0ce73b8481ddce7a3: Status 404 returned error can't find the container with id d6f855d791ce131937389313bfd572bddb2c0b03cdbf65d0ce73b8481ddce7a3
Oct 30 04:07:07.213316 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:07.213210    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2" event={"ID":"0132b0fc-19ea-4401-a7ca-88c3c0728826","Type":"ContainerStarted","Data":"ab0e007ebcccb657d66d21f7e62d1136434766193b8ec660950c64d1ac6c8636"}
Oct 30 04:07:07.213316 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:07.213280    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2" event={"ID":"0132b0fc-19ea-4401-a7ca-88c3c0728826","Type":"ContainerStarted","Data":"d6f855d791ce131937389313bfd572bddb2c0b03cdbf65d0ce73b8481ddce7a3"}
Oct 30 04:07:07.258091 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:07.258027    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2" podStartSLOduration=7.258011387 podStartE2EDuration="7.258011387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:07.253129064 +0000 UTC m=+4291.459309283" watchObservedRunningTime="2026-10-30 04:07:07.258011387 +0000 UTC m=+4291.464191605"
Oct 30 04:07:09.225669 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:09.225634    4494 generic.go:356] "Generic (PLEG): container finished" podID="0132b0fc-19ea-4401-a7ca-88c3c0728826" containerID="ab0e007ebcccb657d66d21f7e62d1136434766193b8ec660950c64d1ac6c8636" exitCode=0
Oct 30 04:07:09.226110 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:09.225708    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2" event={"ID":"0132b0fc-19ea-4401-a7ca-88c3c0728826","Type":"ContainerDied","Data":"ab0e007ebcccb657d66d21f7e62d1136434766193b8ec660950c64d1ac6c8636"}
Oct 30 04:07:10.375724 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:10.375690    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:10.424602 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:10.424569    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"0132b0fc-19ea-4401-a7ca-88c3c0728826\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") "
Oct 30 04:07:10.424808 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:10.424689    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0132b0fc-19ea-4401-a7ca-88c3c0728826-kube-api-access-vhnrp\" (UniqueName: \"kubernetes.io/projected/0132b0fc-19ea-4401-a7ca-88c3c0728826-kube-api-access-vhnrp\") pod \"0132b0fc-19ea-4401-a7ca-88c3c0728826\" (UID: \"0132b0fc-19ea-4401-a7ca-88c3c0728826\") "
Oct 30 04:07:10.427362 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:10.427332    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0132b0fc-19ea-4401-a7ca-88c3c0728826-kube-api-access-vhnrp" pod "0132b0fc-19ea-4401-a7ca-88c3c0728826" (UID: "0132b0fc-19ea-4401-a7ca-88c3c0728826"). InnerVolumeSpecName "kube-api-access-vhnrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:10.427755 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:10.427728    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039" pod "0132b0fc-19ea-4401-a7ca-88c3c0728826" (UID: "0132b0fc-19ea-4401-a7ca-88c3c0728826"). InnerVolumeSpecName "pvc-c7438994-864d-4afe-a05c-ccd8f0458399". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:07:10.526411 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:10.526373    4494 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") on node \"ip-10-0-130-130.ec2.internal\" "
Oct 30 04:07:10.526411 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:10.526413    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhnrp\" (UniqueName: \"kubernetes.io/projected/0132b0fc-19ea-4401-a7ca-88c3c0728826-kube-api-access-vhnrp\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:11.236280 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:11.236250    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"
Oct 30 04:07:11.236472 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:11.236245    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2" event={"ID":"0132b0fc-19ea-4401-a7ca-88c3c0728826","Type":"ContainerDied","Data":"d6f855d791ce131937389313bfd572bddb2c0b03cdbf65d0ce73b8481ddce7a3"}
Oct 30 04:07:11.236472 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:11.236376    4494 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6f855d791ce131937389313bfd572bddb2c0b03cdbf65d0ce73b8481ddce7a3"
Oct 30 04:07:11.445062 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:11.445018    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main" containerID="cri-o://657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0" gracePeriod=45
Oct 30 04:07:11.503425 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:11.503397    4494 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c7438994-864d-4afe-a05c-ccd8f0458399" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039") on node "ip-10-0-130-130.ec2.internal"
Oct 30 04:07:11.535085 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:11.535046    4494 reconciler_common.go:299] "Volume detached for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"csi-db7d3a71899d8761428b694623f5564828c101a903a0d4db5c2f7309332abd93\""
Oct 30 04:07:12.199269 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.199240    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:07:12.241919 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.241862    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7465f30-acd0-4796-bfa6-03d7d2793db4-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7465f30-acd0-4796-bfa6-03d7d2793db4-tls-certs\") pod \"c7465f30-acd0-4796-bfa6-03d7d2793db4\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") "
Oct 30 04:07:12.242120 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.241975    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-tmp-dir\") pod \"c7465f30-acd0-4796-bfa6-03d7d2793db4\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") "
Oct 30 04:07:12.242120 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242016    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-dshm\") pod \"c7465f30-acd0-4796-bfa6-03d7d2793db4\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") "
Oct 30 04:07:12.242120 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242070    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-model-cache\") pod \"c7465f30-acd0-4796-bfa6-03d7d2793db4\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") "
Oct 30 04:07:12.242120 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242102    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7465f30-acd0-4796-bfa6-03d7d2793db4-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7465f30-acd0-4796-bfa6-03d7d2793db4-cabundle-cert\") pod \"c7465f30-acd0-4796-bfa6-03d7d2793db4\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") "
Oct 30 04:07:12.242298 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242197    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7465f30-acd0-4796-bfa6-03d7d2793db4-kube-api-access-4rqt6\" (UniqueName: \"kubernetes.io/projected/c7465f30-acd0-4796-bfa6-03d7d2793db4-kube-api-access-4rqt6\") pod \"c7465f30-acd0-4796-bfa6-03d7d2793db4\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") "
Oct 30 04:07:12.242298 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242227    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-home\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-home\") pod \"c7465f30-acd0-4796-bfa6-03d7d2793db4\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") "
Oct 30 04:07:12.242298 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242275    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-kserve-provision-location\") pod \"c7465f30-acd0-4796-bfa6-03d7d2793db4\" (UID: \"c7465f30-acd0-4796-bfa6-03d7d2793db4\") "
Oct 30 04:07:12.242436 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242306    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-model-cache" pod "c7465f30-acd0-4796-bfa6-03d7d2793db4" (UID: "c7465f30-acd0-4796-bfa6-03d7d2793db4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:12.242550 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242526    4494 generic.go:356] "Generic (PLEG): container finished" podID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerID="657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0" exitCode=0
Oct 30 04:07:12.242618 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242544    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7465f30-acd0-4796-bfa6-03d7d2793db4-cabundle-cert" pod "c7465f30-acd0-4796-bfa6-03d7d2793db4" (UID: "c7465f30-acd0-4796-bfa6-03d7d2793db4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:12.242618 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242597    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.242928 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242639    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" event={"ID":"c7465f30-acd0-4796-bfa6-03d7d2793db4","Type":"ContainerDied","Data":"657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0"}
Oct 30 04:07:12.242928 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242672    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"
Oct 30 04:07:12.242928 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242715    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk" event={"ID":"c7465f30-acd0-4796-bfa6-03d7d2793db4","Type":"ContainerDied","Data":"fdcf029afb3d9c1b911d0ec742127ab7a10510dc91650834dfd8249200252694"}
Oct 30 04:07:12.242928 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.242748    4494 scope.go:122] "RemoveContainer" containerID="657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0"
Oct 30 04:07:12.243398 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.243371    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-home" pod "c7465f30-acd0-4796-bfa6-03d7d2793db4" (UID: "c7465f30-acd0-4796-bfa6-03d7d2793db4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:12.245458 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.245414    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-dshm" pod "c7465f30-acd0-4796-bfa6-03d7d2793db4" (UID: "c7465f30-acd0-4796-bfa6-03d7d2793db4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:12.245458 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.245423    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7465f30-acd0-4796-bfa6-03d7d2793db4-kube-api-access-4rqt6" pod "c7465f30-acd0-4796-bfa6-03d7d2793db4" (UID: "c7465f30-acd0-4796-bfa6-03d7d2793db4"). InnerVolumeSpecName "kube-api-access-4rqt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:12.245649 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.245586    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7465f30-acd0-4796-bfa6-03d7d2793db4-tls-certs" pod "c7465f30-acd0-4796-bfa6-03d7d2793db4" (UID: "c7465f30-acd0-4796-bfa6-03d7d2793db4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:07:12.254005 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.253976    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-kserve-provision-location" pod "c7465f30-acd0-4796-bfa6-03d7d2793db4" (UID: "c7465f30-acd0-4796-bfa6-03d7d2793db4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:12.264379 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.264268    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-tmp-dir" pod "c7465f30-acd0-4796-bfa6-03d7d2793db4" (UID: "c7465f30-acd0-4796-bfa6-03d7d2793db4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:12.274780 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.274750    4494 scope.go:122] "RemoveContainer" containerID="81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51"
Oct 30 04:07:12.284092 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.284066    4494 scope.go:122] "RemoveContainer" containerID="657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0"
Oct 30 04:07:12.284435 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:07:12.284411    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0\": container with ID starting with 657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0 not found: ID does not exist" containerID="657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0"
Oct 30 04:07:12.284500 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.284448    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0"} err="failed to get container status \"657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0\": rpc error: code = NotFound desc = could not find container \"657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0\": container with ID starting with 657e1d1bec40af2fa8b3dda4b3fc4e0675ae661c80ec2af25110cb2b152467f0 not found: ID does not exist"
Oct 30 04:07:12.284500 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.284470    4494 scope.go:122] "RemoveContainer" containerID="81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51"
Oct 30 04:07:12.284761 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:07:12.284744    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51\": container with ID starting with 81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51 not found: ID does not exist" containerID="81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51"
Oct 30 04:07:12.284813 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.284764    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51"} err="failed to get container status \"81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51\": rpc error: code = NotFound desc = could not find container \"81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51\": container with ID starting with 81632a2f36e43cedb345b6254835b483be992af4ef8320cbeb3132c7ffe4cc51 not found: ID does not exist"
Oct 30 04:07:12.343558 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.343512    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7465f30-acd0-4796-bfa6-03d7d2793db4-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.343558 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.343550    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.343558 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.343559    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.343558 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.343567    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7465f30-acd0-4796-bfa6-03d7d2793db4-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.343949 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.343578    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqt6\" (UniqueName: \"kubernetes.io/projected/c7465f30-acd0-4796-bfa6-03d7d2793db4-kube-api-access-4rqt6\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.343949 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.343588    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.343949 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.343599    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7465f30-acd0-4796-bfa6-03d7d2793db4-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:12.590390 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.590353    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"]
Oct 30 04:07:12.602922 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:12.602860    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-f7ff54fd-mm8hk"]
Oct 30 04:07:13.927430 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:13.927395    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" path="/var/lib/kubelet/pods/c7465f30-acd0-4796-bfa6-03d7d2793db4/volumes"
Oct 30 04:07:24.083395 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:07:24.083345    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:07:26.545423 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.545382    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"]
Oct 30 04:07:26.546054 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.545799    4494 cpu_manager.go:414] "containerMap: removing container" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="storage-initializer"
Oct 30 04:07:26.546054 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.545813    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="storage-initializer"
Oct 30 04:07:26.546054 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.545835    4494 cpu_manager.go:414] "containerMap: removing container" podUID="0132b0fc-19ea-4401-a7ca-88c3c0728826" containerName="storage-initializer"
Oct 30 04:07:26.546054 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.545840    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0132b0fc-19ea-4401-a7ca-88c3c0728826" containerName="storage-initializer"
Oct 30 04:07:26.546054 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.545852    4494 cpu_manager.go:414] "containerMap: removing container" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main"
Oct 30 04:07:26.546054 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.545858    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main"
Oct 30 04:07:26.546054 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.545940    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="0132b0fc-19ea-4401-a7ca-88c3c0728826" containerName="storage-initializer"
Oct 30 04:07:26.546054 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.545951    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7465f30-acd0-4796-bfa6-03d7d2793db4" containerName="main"
Oct 30 04:07:26.549357 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.549330    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.565379 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.565348    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"]
Oct 30 04:07:26.566796 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.566778    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Oct 30 04:07:26.567229 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.567212    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Oct 30 04:07:26.576072 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.576039    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Oct 30 04:07:26.576280 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.576095    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-m7n6b\""
Oct 30 04:07:26.676606 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.676564    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-dshm\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.676606 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.676606    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-home\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.676871 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.676724    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkx6q\" (UniqueName: \"kubernetes.io/projected/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-kube-api-access-jkx6q\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.676871 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.676803    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.676871 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.676835    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tls-certs\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.677032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.676888    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-model-cache\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.777816 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.777764    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-model-cache\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.778074 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.777847    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-dshm\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.778074 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.777865    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-home\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.778074 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.777946    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkx6q\" (UniqueName: \"kubernetes.io/projected/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-kube-api-access-jkx6q\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.778074 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.777970    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.778074 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.777995    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tls-certs\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.778333 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.778293    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-model-cache\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.778333 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.778324    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-home\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.778441 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.778406    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.780549 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.780516    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-dshm\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.780786 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.780768    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tls-certs\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:26.803781 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:26.803696    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkx6q\" (UniqueName: \"kubernetes.io/projected/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-kube-api-access-jkx6q\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:36.091355 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:36.091303    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:36.120658 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:36.120618    4494 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:36.192154 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:36.192094    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:36.192370 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:36.192242    4494 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:36.206847 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:36.206814    4494 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") DevicePath \"csi-db7d3a71899d8761428b694623f5564828c101a903a0d4db5c2f7309332abd93\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:36.312422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:36.312388    4494 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b52da1cb97f47b80b1e26f8ba092e9c96c79fbcd1368ddcf17139a7a90451da1/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:36.344664 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:36.344582    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-484pk\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:36.476604 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:36.476560    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-m7n6b\""
Oct 30 04:07:36.482607 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:36.482580    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:36.805216 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:36.805174    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"]
Oct 30 04:07:36.815417 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:07:36.815388    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd750a73d_8cfc_4244_bc4a_1d6c9e647e44.slice/crio-2ce10c9c7851438e7c0f285bc0df0a02f7dfe608a6f6ff59a9f5e97965aa7c4a WatchSource:0}: Error finding container 2ce10c9c7851438e7c0f285bc0df0a02f7dfe608a6f6ff59a9f5e97965aa7c4a: Status 404 returned error can't find the container with id 2ce10c9c7851438e7c0f285bc0df0a02f7dfe608a6f6ff59a9f5e97965aa7c4a
Oct 30 04:07:37.030591 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.030546    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"]
Oct 30 04:07:37.034364 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.034333    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.062278 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.062242    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"]
Oct 30 04:07:37.102359 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.102303    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.102821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.102371    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.102821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.102461    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.102821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.102510    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.102821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.102534    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.102821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.102644    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.102821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.102676    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpn8z\" (UniqueName: \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-kube-api-access-cpn8z\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.102821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.102725    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85914c5-e17a-4115-afe1-b0815bcdad36-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.102821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.102795    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.203353 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203313    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.203353 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203364    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.203645 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203381    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpn8z\" (UniqueName: \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-kube-api-access-cpn8z\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.203645 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203407    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85914c5-e17a-4115-afe1-b0815bcdad36-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.203645 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203435    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.203645 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203476    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.203645 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203509    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.203645 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203534    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.203645 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203582    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.204071 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203790    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.204071 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203811    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.204071 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203877    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.204071 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.203972    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.204293 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.204219    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85914c5-e17a-4115-afe1-b0815bcdad36-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.206321 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.206293    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.206563 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.206544    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.227527 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.227486    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.227758 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.227730    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpn8z\" (UniqueName: \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-kube-api-access-cpn8z\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.348946 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.348829    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:37.375795 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.375754    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" event={"ID":"d750a73d-8cfc-4244-bc4a-1d6c9e647e44","Type":"ContainerStarted","Data":"adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314"}
Oct 30 04:07:37.376050 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.375815    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" event={"ID":"d750a73d-8cfc-4244-bc4a-1d6c9e647e44","Type":"ContainerStarted","Data":"2ce10c9c7851438e7c0f285bc0df0a02f7dfe608a6f6ff59a9f5e97965aa7c4a"}
Oct 30 04:07:37.422545 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.422493    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" podStartSLOduration=11.422478207 podStartE2EDuration="11.422478207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:37.422177522 +0000 UTC m=+4321.628357740" watchObservedRunningTime="2026-10-30 04:07:37.422478207 +0000 UTC m=+4321.628658425"
Oct 30 04:07:37.483691 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.483645    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:37.656395 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.656350    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"]
Oct 30 04:07:37.656667 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:07:37.656642    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85914c5_e17a_4115_afe1_b0815bcdad36.slice/crio-be584af147691ba5530af6cc8abefe92e2c67327df7c767429bf5d639915d4ec WatchSource:0}: Error finding container be584af147691ba5530af6cc8abefe92e2c67327df7c767429bf5d639915d4ec: Status 404 returned error can't find the container with id be584af147691ba5530af6cc8abefe92e2c67327df7c767429bf5d639915d4ec
Oct 30 04:07:37.662587 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.662546    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:07:37.662709 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.662631    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:07:37.662709 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:37.662660    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:07:38.408521 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:38.405706    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" event={"ID":"d85914c5-e17a-4115-afe1-b0815bcdad36","Type":"ContainerStarted","Data":"a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7"}
Oct 30 04:07:38.408521 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:38.405807    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" event={"ID":"d85914c5-e17a-4115-afe1-b0815bcdad36","Type":"ContainerStarted","Data":"be584af147691ba5530af6cc8abefe92e2c67327df7c767429bf5d639915d4ec"}
Oct 30 04:07:38.465263 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:38.465205    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" podStartSLOduration=1.465189279 podStartE2EDuration="1.465189279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:07:38.463027 +0000 UTC m=+4322.669207243" watchObservedRunningTime="2026-10-30 04:07:38.465189279 +0000 UTC m=+4322.671369496"
Oct 30 04:07:39.349509 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:39.349470    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:39.350873 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:39.350842    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 04:07:39.351022 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:39.350914    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 04:07:40.349662 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:40.349617    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 04:07:40.350145 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:40.349680    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 04:07:41.349817 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:41.349773    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 04:07:41.350294 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:41.349831    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 04:07:42.349284 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:42.349232    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 04:07:42.349529 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:42.349298    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 04:07:43.349918 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:43.349868    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Oct 30 04:07:43.350387 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:43.349947    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Oct 30 04:07:44.354080 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:44.354049    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:44.354600 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:44.354354    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:44.355319 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:44.355300    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:07:46.482808 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:46.482757    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:07:46.484370 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:46.484341    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Oct 30 04:07:51.563426 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:51.563379    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"]
Oct 30 04:07:51.563989 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:51.563678    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy" containerID="cri-o://04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07" gracePeriod=30
Oct 30 04:07:52.622415 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:52.622357    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:07:54.085091 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:07:54.085038    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:07:56.483595 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.483545    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Oct 30 04:07:56.819507 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.819474    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:07:56.902517 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.902478    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-token\" (UniqueName: \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-token\") pod \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") "
Oct 30 04:07:56.902753 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.902596    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-kube-api-access-8cq6n\" (UniqueName: \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-kube-api-access-8cq6n\") pod \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") "
Oct 30 04:07:56.902753 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.902645    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-data\") pod \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") "
Oct 30 04:07:56.902753 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.902675    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istiod-ca-cert\") pod \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") "
Oct 30 04:07:56.902753 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.902737    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-envoy\") pod \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") "
Oct 30 04:07:56.903002 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.902800    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-certs\") pod \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") "
Oct 30 04:07:56.903002 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.902829    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-credential-socket\") pod \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") "
Oct 30 04:07:56.903002 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.902893    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-socket\") pod \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") "
Oct 30 04:07:56.903002 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.902932    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-data" pod "54801ec4-fab8-4c3e-befb-2b0b16d74f9e" (UID: "54801ec4-fab8-4c3e-befb-2b0b16d74f9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:56.903002 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.902949    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-podinfo\") pod \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\" (UID: \"54801ec4-fab8-4c3e-befb-2b0b16d74f9e\") "
Oct 30 04:07:56.903924 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.903519    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-certs" pod "54801ec4-fab8-4c3e-befb-2b0b16d74f9e" (UID: "54801ec4-fab8-4c3e-befb-2b0b16d74f9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:56.903924 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.903550    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:56.904395 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.904269    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istiod-ca-cert" pod "54801ec4-fab8-4c3e-befb-2b0b16d74f9e" (UID: "54801ec4-fab8-4c3e-befb-2b0b16d74f9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:07:56.904492 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.904385    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-credential-socket" pod "54801ec4-fab8-4c3e-befb-2b0b16d74f9e" (UID: "54801ec4-fab8-4c3e-befb-2b0b16d74f9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:56.904492 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.904458    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-socket" pod "54801ec4-fab8-4c3e-befb-2b0b16d74f9e" (UID: "54801ec4-fab8-4c3e-befb-2b0b16d74f9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:56.905543 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.905511    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-podinfo" pod "54801ec4-fab8-4c3e-befb-2b0b16d74f9e" (UID: "54801ec4-fab8-4c3e-befb-2b0b16d74f9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:07:56.905657 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.905631    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-kube-api-access-8cq6n" pod "54801ec4-fab8-4c3e-befb-2b0b16d74f9e" (UID: "54801ec4-fab8-4c3e-befb-2b0b16d74f9e"). InnerVolumeSpecName "kube-api-access-8cq6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:56.905726 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.905684    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-token" pod "54801ec4-fab8-4c3e-befb-2b0b16d74f9e" (UID: "54801ec4-fab8-4c3e-befb-2b0b16d74f9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:07:56.906405 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:56.906383    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-envoy" pod "54801ec4-fab8-4c3e-befb-2b0b16d74f9e" (UID: "54801ec4-fab8-4c3e-befb-2b0b16d74f9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:07:57.004400 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.004356    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cq6n\" (UniqueName: \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-kube-api-access-8cq6n\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:57.004400 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.004396    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:57.004400 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.004407    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:57.004676 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.004415    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:57.004676 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.004423    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:57.004676 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.004432    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:57.004676 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.004441    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:57.004676 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.004453    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54801ec4-fab8-4c3e-befb-2b0b16d74f9e-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:07:57.501915 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.501859    4494 generic.go:356] "Generic (PLEG): container finished" podID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerID="04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07" exitCode=0
Oct 30 04:07:57.502407 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.501925    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"
Oct 30 04:07:57.502407 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.501944    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" event={"ID":"54801ec4-fab8-4c3e-befb-2b0b16d74f9e","Type":"ContainerDied","Data":"04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07"}
Oct 30 04:07:57.502407 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.502003    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5" event={"ID":"54801ec4-fab8-4c3e-befb-2b0b16d74f9e","Type":"ContainerDied","Data":"adc031fa7edfc8fd5085d19748b995040ae11c96c71c8771044171cad1fd6149"}
Oct 30 04:07:57.502407 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.502029    4494 scope.go:122] "RemoveContainer" containerID="04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07"
Oct 30 04:07:57.512187 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.512161    4494 scope.go:122] "RemoveContainer" containerID="52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a"
Oct 30 04:07:57.521545 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.521522    4494 scope.go:122] "RemoveContainer" containerID="04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07"
Oct 30 04:07:57.521821 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:07:57.521800    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07\": container with ID starting with 04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07 not found: ID does not exist" containerID="04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07"
Oct 30 04:07:57.521890 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.521832    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07"} err="failed to get container status \"04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07\": rpc error: code = NotFound desc = could not find container \"04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07\": container with ID starting with 04faa2a0e0da97e0ff40cee5785b4e6ebba5c0ea4a5c245e20037e67a1008a07 not found: ID does not exist"
Oct 30 04:07:57.521890 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.521851    4494 scope.go:122] "RemoveContainer" containerID="52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a"
Oct 30 04:07:57.522166 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:07:57.522137    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a\": container with ID starting with 52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a not found: ID does not exist" containerID="52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a"
Oct 30 04:07:57.522206 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.522179    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a"} err="failed to get container status \"52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a\": rpc error: code = NotFound desc = could not find container \"52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a\": container with ID starting with 52551efaafcf2ca8d35b3a89199bac89d0ab24ce8f5e15030f45cb205091745a not found: ID does not exist"
Oct 30 04:07:57.566614 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.566573    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"]
Oct 30 04:07:57.570197 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.570168    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7bd678d5fc-xk4x5"]
Oct 30 04:07:57.927360 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:07:57.927314    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" path="/var/lib/kubelet/pods/54801ec4-fab8-4c3e-befb-2b0b16d74f9e/volumes"
Oct 30 04:08:06.483202 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:08:06.483127    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Oct 30 04:08:16.483127 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:08:16.483083    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Oct 30 04:08:24.084198 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:08:24.084157    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:08:26.483976 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:08:26.483934    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Oct 30 04:08:36.483172 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:08:36.483118    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Oct 30 04:08:46.483827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:08:46.483777    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": dial tcp 10.134.0.87:8000: connect: connection refused"
Oct 30 04:08:54.083585 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:08:54.083537    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:08:56.493681 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:08:56.493646    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:08:56.501733 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:08:56.501704    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:09:24.082360 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:09:24.082321    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:09:24.267662 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:24.267628    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-55d4fd88bc-484pk_d750a73d-8cfc-4244-bc4a-1d6c9e647e44/main/0.log"
Oct 30 04:09:27.572515 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:27.572477    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"]
Oct 30 04:09:42.596849 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:42.596798    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main" containerID="cri-o://adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314" gracePeriod=45
Oct 30 04:09:43.251660 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.251631    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:09:43.324452 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.324365    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-home\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-home\") pod \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") "
Oct 30 04:09:43.324452 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.324429    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-dshm\") pod \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") "
Oct 30 04:09:43.324452 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.324449    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tmp-dir\") pod \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") "
Oct 30 04:09:43.324999 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.324475    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tls-certs\" (UniqueName: \"kubernetes.io/secret/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tls-certs\") pod \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") "
Oct 30 04:09:43.324999 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.324530    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-kube-api-access-jkx6q\" (UniqueName: \"kubernetes.io/projected/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-kube-api-access-jkx6q\") pod \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") "
Oct 30 04:09:43.324999 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.324575    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-model-cache\") pod \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") "
Oct 30 04:09:43.324999 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.324681    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") pod \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\" (UID: \"d750a73d-8cfc-4244-bc4a-1d6c9e647e44\") "
Oct 30 04:09:43.324999 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.324949    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-model-cache" pod "d750a73d-8cfc-4244-bc4a-1d6c9e647e44" (UID: "d750a73d-8cfc-4244-bc4a-1d6c9e647e44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:09:43.324999 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.324834    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-home" pod "d750a73d-8cfc-4244-bc4a-1d6c9e647e44" (UID: "d750a73d-8cfc-4244-bc4a-1d6c9e647e44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:09:43.327019 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.326980    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-dshm" pod "d750a73d-8cfc-4244-bc4a-1d6c9e647e44" (UID: "d750a73d-8cfc-4244-bc4a-1d6c9e647e44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:09:43.327152 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.327017    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-kube-api-access-jkx6q" pod "d750a73d-8cfc-4244-bc4a-1d6c9e647e44" (UID: "d750a73d-8cfc-4244-bc4a-1d6c9e647e44"). InnerVolumeSpecName "kube-api-access-jkx6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:09:43.327152 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.327060    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tls-certs" pod "d750a73d-8cfc-4244-bc4a-1d6c9e647e44" (UID: "d750a73d-8cfc-4244-bc4a-1d6c9e647e44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:09:43.334289 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.334262    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039" pod "d750a73d-8cfc-4244-bc4a-1d6c9e647e44" (UID: "d750a73d-8cfc-4244-bc4a-1d6c9e647e44"). InnerVolumeSpecName "pvc-c7438994-864d-4afe-a05c-ccd8f0458399". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 04:09:43.338209 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.338182    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tmp-dir" pod "d750a73d-8cfc-4244-bc4a-1d6c9e647e44" (UID: "d750a73d-8cfc-4244-bc4a-1d6c9e647e44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:09:43.425442 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.425406    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkx6q\" (UniqueName: \"kubernetes.io/projected/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-kube-api-access-jkx6q\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:09:43.425442 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.425436    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:09:43.425643 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.425469    4494 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") on node \"ip-10-0-130-130.ec2.internal\" "
Oct 30 04:09:43.425643 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.425484    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:09:43.425643 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.425496    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:09:43.425643 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.425508    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:09:43.425643 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.425519    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d750a73d-8cfc-4244-bc4a-1d6c9e647e44-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:09:43.455266 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.455246    4494 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c7438994-864d-4afe-a05c-ccd8f0458399" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039") on node "ip-10-0-130-130.ec2.internal"
Oct 30 04:09:43.526813 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:43.526778    4494 reconciler_common.go:299] "Volume detached for volume \"pvc-c7438994-864d-4afe-a05c-ccd8f0458399\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06fcfcc492a701039\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"csi-db7d3a71899d8761428b694623f5564828c101a903a0d4db5c2f7309332abd93\""
Oct 30 04:09:44.030344 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:44.030303    4494 generic.go:356] "Generic (PLEG): container finished" podID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerID="adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314" exitCode=0
Oct 30 04:09:44.030782 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:44.030381    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"
Oct 30 04:09:44.030782 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:44.030387    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" event={"ID":"d750a73d-8cfc-4244-bc4a-1d6c9e647e44","Type":"ContainerDied","Data":"adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314"}
Oct 30 04:09:44.030782 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:44.030446    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk" event={"ID":"d750a73d-8cfc-4244-bc4a-1d6c9e647e44","Type":"ContainerDied","Data":"2ce10c9c7851438e7c0f285bc0df0a02f7dfe608a6f6ff59a9f5e97965aa7c4a"}
Oct 30 04:09:44.030782 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:44.030471    4494 scope.go:122] "RemoveContainer" containerID="adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314"
Oct 30 04:09:44.039294 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:44.039270    4494 scope.go:122] "RemoveContainer" containerID="adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314"
Oct 30 04:09:44.039573 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:09:44.039552    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314\": container with ID starting with adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314 not found: ID does not exist" containerID="adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314"
Oct 30 04:09:44.039626 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:44.039582    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314"} err="failed to get container status \"adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314\": rpc error: code = NotFound desc = could not find container \"adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314\": container with ID starting with adff024928b45e60dab0776eb8e423ba62d5a8a3cf23e7125d534818c650c314 not found: ID does not exist"
Oct 30 04:09:44.094654 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:44.094623    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"]
Oct 30 04:09:44.107206 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:44.107174    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-484pk"]
Oct 30 04:09:45.927304 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:45.927267    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" path="/var/lib/kubelet/pods/d750a73d-8cfc-4244-bc4a-1d6c9e647e44/volumes"
Oct 30 04:09:48.657744 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.657708    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"]
Oct 30 04:09:48.658330 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.658308    4494 cpu_manager.go:414] "containerMap: removing container" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy"
Oct 30 04:09:48.658408 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.658333    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy"
Oct 30 04:09:48.658408 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.658365    4494 cpu_manager.go:414] "containerMap: removing container" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy"
Oct 30 04:09:48.658408 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.658375    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy"
Oct 30 04:09:48.658408 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.658405    4494 cpu_manager.go:414] "containerMap: removing container" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main"
Oct 30 04:09:48.658600 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.658413    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main"
Oct 30 04:09:48.658600 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.658499    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy"
Oct 30 04:09:48.658600 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.658514    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d750a73d-8cfc-4244-bc4a-1d6c9e647e44" containerName="main"
Oct 30 04:09:48.661567 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.661544    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.681230 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.681200    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"kube-root-ca.crt\""
Oct 30 04:09:48.681725 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.681696    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"openshift-gw-ca-root-cert\""
Oct 30 04:09:48.692410 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.692376    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"model-routing-only-gateway-openshift-default-dockercfg-6p5nz\""
Oct 30 04:09:48.693582 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.693556    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"]
Oct 30 04:09:48.781049 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.781011    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"openshift-service-ca.crt\""
Oct 30 04:09:48.783412 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.783387    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-data\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.783484 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.783424    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-socket\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.783539 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.783505    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9301511d-3974-4f95-81c3-c41063623aeb-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.783539 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.783531    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-credential-socket\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.783618 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.783553    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-certs\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.783658 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.783616    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9301511d-3974-4f95-81c3-c41063623aeb-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.783717 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.783667    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.783795 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.783776    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49hjc\" (UniqueName: \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-kube-api-access-49hjc\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.783852 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.783809    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-istio-token\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885157 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885058    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9301511d-3974-4f95-81c3-c41063623aeb-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885157 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885100    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885157 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885133    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hjc\" (UniqueName: \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-kube-api-access-49hjc\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885157 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885150    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-istio-token\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885539 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885193    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-data\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885539 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885223    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-socket\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885539 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885271    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9301511d-3974-4f95-81c3-c41063623aeb-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885539 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885301    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-credential-socket\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885539 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885332    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-certs\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885872 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885725    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-socket\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885872 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885768    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-certs\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.885872 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885811    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-data\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.886237 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885960    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-credential-socket\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.886237 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.885989    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9301511d-3974-4f95-81c3-c41063623aeb-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.887875 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.887852    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.888129 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.888109    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9301511d-3974-4f95-81c3-c41063623aeb-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.912725 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.912689    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-istio-token\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.912855 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.912694    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hjc\" (UniqueName: \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-kube-api-access-49hjc\") pod \"model-routing-only-gateway-openshift-default-5f658554c7-b8wt2\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:48.974863 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:48.974826    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:49.302217 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:49.302185    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"]
Oct 30 04:09:49.302965 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:09:49.302934    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9301511d_3974_4f95_81c3_c41063623aeb.slice/crio-2d713c188a50b39a277e23cb01107fb4374848dc2b96d73e9baa4d84aa561d60 WatchSource:0}: Error finding container 2d713c188a50b39a277e23cb01107fb4374848dc2b96d73e9baa4d84aa561d60: Status 404 returned error can't find the container with id 2d713c188a50b39a277e23cb01107fb4374848dc2b96d73e9baa4d84aa561d60
Oct 30 04:09:49.305161 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:49.305129    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:09:49.305221 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:49.305196    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:09:49.305262 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:49.305227    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:09:50.061503 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:50.061456    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" event={"ID":"9301511d-3974-4f95-81c3-c41063623aeb","Type":"ContainerStarted","Data":"0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578"}
Oct 30 04:09:50.061931 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:50.061534    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" event={"ID":"9301511d-3974-4f95-81c3-c41063623aeb","Type":"ContainerStarted","Data":"2d713c188a50b39a277e23cb01107fb4374848dc2b96d73e9baa4d84aa561d60"}
Oct 30 04:09:50.122998 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:50.122933    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" podStartSLOduration=2.122917097 podStartE2EDuration="2.122917097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:09:50.121344302 +0000 UTC m=+4454.327524523" watchObservedRunningTime="2026-10-30 04:09:50.122917097 +0000 UTC m=+4454.329097309"
Oct 30 04:09:50.975310 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:50.975268    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:50.980343 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:50.980317    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:51.065855 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:51.065825    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:51.067102 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:51.067074    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:09:52.932772 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:52.932736    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"]
Oct 30 04:09:52.937422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:52.937395    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-pjtw2"]
Oct 30 04:09:53.927395 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:09:53.927357    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0132b0fc-19ea-4401-a7ca-88c3c0728826" path="/var/lib/kubelet/pods/0132b0fc-19ea-4401-a7ca-88c3c0728826/volumes"
Oct 30 04:09:54.082532 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:09:54.082494    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:10:08.662730 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.662692    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"]
Oct 30 04:10:08.663392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.663315    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="54801ec4-fab8-4c3e-befb-2b0b16d74f9e" containerName="istio-proxy"
Oct 30 04:10:08.667259 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.667230    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.694340 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.694312    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"default-dockercfg-gdspd\""
Oct 30 04:10:08.694340 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.694321    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-f772d59b\"/\"model-routing-only-7aad4d71-kserve-self-signed-certs\""
Oct 30 04:10:08.697265 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.697239    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"]
Oct 30 04:10:08.770399 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.770351    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-home\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.770637 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.770433    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-tmp-dir\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.770637 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.770565    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-dshm\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.770637 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.770618    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhgkk\" (UniqueName: \"kubernetes.io/projected/5b477960-9d27-4dbc-9bc9-add990d40f78-kube-api-access-xhgkk\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.770779 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.770687    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b477960-9d27-4dbc-9bc9-add990d40f78-tls-certs\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.770779 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.770741    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-model-cache\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.872086 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.872032    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-dshm\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.872324 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.872102    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhgkk\" (UniqueName: \"kubernetes.io/projected/5b477960-9d27-4dbc-9bc9-add990d40f78-kube-api-access-xhgkk\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.872324 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.872130    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b477960-9d27-4dbc-9bc9-add990d40f78-tls-certs\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.872324 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.872178    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-model-cache\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.872324 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.872215    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-home\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.872324 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.872268    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-tmp-dir\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.872650 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.872619    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-model-cache\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.872767 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.872658    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-home\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.872767 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.872718    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-tmp-dir\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.874760 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.874735    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-dshm\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.874955 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.874937    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b477960-9d27-4dbc-9bc9-add990d40f78-tls-certs\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.896179 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.896144    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhgkk\" (UniqueName: \"kubernetes.io/projected/5b477960-9d27-4dbc-9bc9-add990d40f78-kube-api-access-xhgkk\") pod \"model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") " pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:08.984832 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:08.984720    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:09.285119 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:10:09.285086    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b477960_9d27_4dbc_9bc9_add990d40f78.slice/crio-66c2ecdef2ff2a6b4a499cf68f11c7c9cb7fdebdc12f5394dd958bee7d47da38 WatchSource:0}: Error finding container 66c2ecdef2ff2a6b4a499cf68f11c7c9cb7fdebdc12f5394dd958bee7d47da38: Status 404 returned error can't find the container with id 66c2ecdef2ff2a6b4a499cf68f11c7c9cb7fdebdc12f5394dd958bee7d47da38
Oct 30 04:10:09.285244 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:09.285205    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"]
Oct 30 04:10:10.151257 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:10.151217    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq" event={"ID":"5b477960-9d27-4dbc-9bc9-add990d40f78","Type":"ContainerStarted","Data":"8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41"}
Oct 30 04:10:10.151695 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:10.151280    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq" event={"ID":"5b477960-9d27-4dbc-9bc9-add990d40f78","Type":"ContainerStarted","Data":"66c2ecdef2ff2a6b4a499cf68f11c7c9cb7fdebdc12f5394dd958bee7d47da38"}
Oct 30 04:10:10.193504 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:10.193445    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq" podStartSLOduration=2.193429421 podStartE2EDuration="2.193429421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:10.192933447 +0000 UTC m=+4474.399113662" watchObservedRunningTime="2026-10-30 04:10:10.193429421 +0000 UTC m=+4474.399609638"
Oct 30 04:10:10.994064 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:10.994012    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:18.984943 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:18.984881    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:18.998246 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:18.998211    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:19.195231 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:19.195195    4494 generic.go:356] "Generic (PLEG): container finished" podID="9301511d-3974-4f95-81c3-c41063623aeb" containerID="0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578" exitCode=0
Oct 30 04:10:19.195460 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:19.195271    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" event={"ID":"9301511d-3974-4f95-81c3-c41063623aeb","Type":"ContainerDied","Data":"0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578"}
Oct 30 04:10:19.195881 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:19.195852    4494 scope.go:122] "RemoveContainer" containerID="0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578"
Oct 30 04:10:19.197068 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:19.197037    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:10:19.197165 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:19.197118    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:10:19.197165 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:19.197158    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:10:19.207393 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:19.207369    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:20.210150 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:20.210112    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" event={"ID":"9301511d-3974-4f95-81c3-c41063623aeb","Type":"ContainerStarted","Data":"ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d"}
Oct 30 04:10:20.975119 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:20.975059    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:10:20.976291 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:20.976258    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 04:10:21.065783 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:21.065742    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:10:21.976397 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:21.976288    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 04:10:22.975599 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:22.975552    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 04:10:23.975727 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:23.975687    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 04:10:24.105627 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:10:24.105585    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:10:24.975503 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:24.975457    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 30 04:10:25.979388 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:25.979352    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:10:25.980323 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:25.980303    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:10:32.526873 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.526825    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 04:10:32.530878 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.530853    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.557319 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.557281    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 04:10:32.590869 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.590837    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Oct 30 04:10:32.591209 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.591185    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-b9z9s\""
Oct 30 04:10:32.591287 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.591185    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-8cr9j\""
Oct 30 04:10:32.591287 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.591195    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Oct 30 04:10:32.591867 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.591850    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Oct 30 04:10:32.591976 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.591959    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Oct 30 04:10:32.592022 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.592014    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:10:32.605652 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.605623    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.605779 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.605668    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5988729d-daeb-4532-9716-9a464840c988-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.605820 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.605778    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.605820 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.605813    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5988729d-daeb-4532-9716-9a464840c988-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.605923 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.605848    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.605923 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.605871    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k4nf\" (UniqueName: \"kubernetes.io/projected/5988729d-daeb-4532-9716-9a464840c988-kube-api-access-8k4nf\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.606019 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.605968    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.606019 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.605989    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707131 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707090    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707131 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707138    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5988729d-daeb-4532-9716-9a464840c988-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707162    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707179    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k4nf\" (UniqueName: \"kubernetes.io/projected/5988729d-daeb-4532-9716-9a464840c988-kube-api-access-8k4nf\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707317    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707356    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707403    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707713 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707456    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5988729d-daeb-4532-9716-9a464840c988-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707713 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707564    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707713 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707635    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707843 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707782    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.707935 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.707841    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.708057 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.708037    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5988729d-daeb-4532-9716-9a464840c988-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.709812 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.709786    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.709957 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.709820    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5988729d-daeb-4532-9716-9a464840c988-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.736459 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.736426    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k4nf\" (UniqueName: \"kubernetes.io/projected/5988729d-daeb-4532-9716-9a464840c988-kube-api-access-8k4nf\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:32.849025 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:32.848924    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:10:33.166892 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:33.166853    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 04:10:33.167198 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:10:33.167172    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5988729d_daeb_4532_9716_9a464840c988.slice/crio-5f3a602bd4eae7099743766e0fb17224c4fedad633f371850feb38902ea4f38a WatchSource:0}: Error finding container 5f3a602bd4eae7099743766e0fb17224c4fedad633f371850feb38902ea4f38a: Status 404 returned error can't find the container with id 5f3a602bd4eae7099743766e0fb17224c4fedad633f371850feb38902ea4f38a
Oct 30 04:10:33.272136 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:33.272089    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5988729d-daeb-4532-9716-9a464840c988","Type":"ContainerStarted","Data":"efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4"}
Oct 30 04:10:33.272355 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:33.272165    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5988729d-daeb-4532-9716-9a464840c988","Type":"ContainerStarted","Data":"5f3a602bd4eae7099743766e0fb17224c4fedad633f371850feb38902ea4f38a"}
Oct 30 04:10:36.289422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:36.289383    4494 generic.go:356] "Generic (PLEG): container finished" podID="5988729d-daeb-4532-9716-9a464840c988" containerID="efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4" exitCode=0
Oct 30 04:10:36.289827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:36.289452    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5988729d-daeb-4532-9716-9a464840c988","Type":"ContainerDied","Data":"efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4"}
Oct 30 04:10:36.784682 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:36.784650    4494 scope.go:122] "RemoveContainer" containerID="838c7cc9c2efe281be6cb253fa13e831d477f1fc4b5d11c9f5793e10198cc001"
Oct 30 04:10:37.296656 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:37.296612    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5988729d-daeb-4532-9716-9a464840c988","Type":"ContainerStarted","Data":"e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2"}
Oct 30 04:10:37.367854 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:37.367790    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.03862027 podStartE2EDuration="5.367774015s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.329153745s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:10:37.354548257 +0000 UTC m=+4501.560728476" watchObservedRunningTime="2026-10-30 04:10:37.367774015 +0000 UTC m=+4501.573954233"
Oct 30 04:10:53.928065 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:53.928032    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"]
Oct 30 04:10:53.936552 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:53.936516    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"]
Oct 30 04:10:53.936863 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:53.936827    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy" containerID="cri-o://ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d" gracePeriod=30
Oct 30 04:10:53.938675 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:10:53.938637    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41" cmd=["/bin/sleep","15"]
Oct 30 04:10:53.938799 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:53.938702    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 04:10:53.938799 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:10:53.938722    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq" podUID="5b477960-9d27-4dbc-9bc9-add990d40f78" containerName="main" containerID="cri-o://8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41"
Oct 30 04:10:53.938799 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:53.938780    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq" podUID="5b477960-9d27-4dbc-9bc9-add990d40f78" containerName="main" containerID="cri-o://8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41" gracePeriod=60
Oct 30 04:10:54.083180 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:10:54.083141    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:10:54.196429 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.196338    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq" podUID="5b477960-9d27-4dbc-9bc9-add990d40f78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": dial tcp 10.134.0.90:8000: connect: connection refused"
Oct 30 04:10:54.302678 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.302650    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:54.388152 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.388113    4494 generic.go:356] "Generic (PLEG): container finished" podID="5b477960-9d27-4dbc-9bc9-add990d40f78" containerID="8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41" exitCode=0
Oct 30 04:10:54.388326 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.388169    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"
Oct 30 04:10:54.388471 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.388168    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq" event={"ID":"5b477960-9d27-4dbc-9bc9-add990d40f78","Type":"ContainerDied","Data":"8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41"}
Oct 30 04:10:54.388517 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.388503    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq" event={"ID":"5b477960-9d27-4dbc-9bc9-add990d40f78","Type":"ContainerDied","Data":"66c2ecdef2ff2a6b4a499cf68f11c7c9cb7fdebdc12f5394dd958bee7d47da38"}
Oct 30 04:10:54.388553 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.388531    4494 scope.go:122] "RemoveContainer" containerID="8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41"
Oct 30 04:10:54.398630 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.398605    4494 scope.go:122] "RemoveContainer" containerID="8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41"
Oct 30 04:10:54.398987 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:10:54.398964    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41\": container with ID starting with 8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41 not found: ID does not exist" containerID="8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41"
Oct 30 04:10:54.399049 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.398996    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41"} err="failed to get container status \"8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41\": rpc error: code = NotFound desc = could not find container \"8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41\": container with ID starting with 8cc9a7fb824511222337309c644ef48dc99888757ac990756906972bc3d9db41 not found: ID does not exist"
Oct 30 04:10:54.423583 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.423543    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b477960-9d27-4dbc-9bc9-add990d40f78-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b477960-9d27-4dbc-9bc9-add990d40f78-tls-certs\") pod \"5b477960-9d27-4dbc-9bc9-add990d40f78\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") "
Oct 30 04:10:54.423752 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.423595    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b477960-9d27-4dbc-9bc9-add990d40f78-kube-api-access-xhgkk\" (UniqueName: \"kubernetes.io/projected/5b477960-9d27-4dbc-9bc9-add990d40f78-kube-api-access-xhgkk\") pod \"5b477960-9d27-4dbc-9bc9-add990d40f78\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") "
Oct 30 04:10:54.423752 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.423655    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-tmp-dir\") pod \"5b477960-9d27-4dbc-9bc9-add990d40f78\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") "
Oct 30 04:10:54.423752 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.423685    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-dshm\") pod \"5b477960-9d27-4dbc-9bc9-add990d40f78\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") "
Oct 30 04:10:54.423752 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.423716    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-home\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-home\") pod \"5b477960-9d27-4dbc-9bc9-add990d40f78\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") "
Oct 30 04:10:54.424002 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.423751    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-model-cache\") pod \"5b477960-9d27-4dbc-9bc9-add990d40f78\" (UID: \"5b477960-9d27-4dbc-9bc9-add990d40f78\") "
Oct 30 04:10:54.424130 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.424094    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-tmp-dir" pod "5b477960-9d27-4dbc-9bc9-add990d40f78" (UID: "5b477960-9d27-4dbc-9bc9-add990d40f78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:54.424130 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.424109    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-home" pod "5b477960-9d27-4dbc-9bc9-add990d40f78" (UID: "5b477960-9d27-4dbc-9bc9-add990d40f78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:54.424273 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.424259    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-model-cache" pod "5b477960-9d27-4dbc-9bc9-add990d40f78" (UID: "5b477960-9d27-4dbc-9bc9-add990d40f78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:54.426213 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.426190    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b477960-9d27-4dbc-9bc9-add990d40f78-kube-api-access-xhgkk" pod "5b477960-9d27-4dbc-9bc9-add990d40f78" (UID: "5b477960-9d27-4dbc-9bc9-add990d40f78"). InnerVolumeSpecName "kube-api-access-xhgkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:54.426484 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.426462    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-dshm" pod "5b477960-9d27-4dbc-9bc9-add990d40f78" (UID: "5b477960-9d27-4dbc-9bc9-add990d40f78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:54.426566 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.426466    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b477960-9d27-4dbc-9bc9-add990d40f78-tls-certs" pod "5b477960-9d27-4dbc-9bc9-add990d40f78" (UID: "5b477960-9d27-4dbc-9bc9-add990d40f78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:10:54.525755 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.525657    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:54.525755 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.525703    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:54.525755 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.525716    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:54.525755 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.525729    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b477960-9d27-4dbc-9bc9-add990d40f78-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:54.525755 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.525743    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b477960-9d27-4dbc-9bc9-add990d40f78-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:54.525755 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.525756    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhgkk\" (UniqueName: \"kubernetes.io/projected/5b477960-9d27-4dbc-9bc9-add990d40f78-kube-api-access-xhgkk\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:54.740935 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.740880    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"]
Oct 30 04:10:54.748334 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:54.748302    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-7aad4d71-kserve-6557db66dd-nmfxq"]
Oct 30 04:10:55.929684 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:55.929641    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b477960-9d27-4dbc-9bc9-add990d40f78" path="/var/lib/kubelet/pods/5b477960-9d27-4dbc-9bc9-add990d40f78/volumes"
Oct 30 04:10:55.982868 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:55.982816    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:10:59.201613 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.201535    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:10:59.271186 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271150    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-credential-socket\") pod \"9301511d-3974-4f95-81c3-c41063623aeb\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") "
Oct 30 04:10:59.271369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271208    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-istio-token\" (UniqueName: \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-istio-token\") pod \"9301511d-3974-4f95-81c3-c41063623aeb\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") "
Oct 30 04:10:59.271369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271249    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-certs\") pod \"9301511d-3974-4f95-81c3-c41063623aeb\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") "
Oct 30 04:10:59.271369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271291    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-envoy\") pod \"9301511d-3974-4f95-81c3-c41063623aeb\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") "
Oct 30 04:10:59.271369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271320    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-kube-api-access-49hjc\" (UniqueName: \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-kube-api-access-49hjc\") pod \"9301511d-3974-4f95-81c3-c41063623aeb\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") "
Oct 30 04:10:59.271369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271348    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-socket\") pod \"9301511d-3974-4f95-81c3-c41063623aeb\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") "
Oct 30 04:10:59.271369 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271366    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9301511d-3974-4f95-81c3-c41063623aeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9301511d-3974-4f95-81c3-c41063623aeb-istiod-ca-cert\") pod \"9301511d-3974-4f95-81c3-c41063623aeb\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") "
Oct 30 04:10:59.271705 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271444    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-data\") pod \"9301511d-3974-4f95-81c3-c41063623aeb\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") "
Oct 30 04:10:59.271705 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271471    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9301511d-3974-4f95-81c3-c41063623aeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9301511d-3974-4f95-81c3-c41063623aeb-istio-podinfo\") pod \"9301511d-3974-4f95-81c3-c41063623aeb\" (UID: \"9301511d-3974-4f95-81c3-c41063623aeb\") "
Oct 30 04:10:59.271705 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271516    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-credential-socket" pod "9301511d-3974-4f95-81c3-c41063623aeb" (UID: "9301511d-3974-4f95-81c3-c41063623aeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:59.271705 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271646    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-certs" pod "9301511d-3974-4f95-81c3-c41063623aeb" (UID: "9301511d-3974-4f95-81c3-c41063623aeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:59.271938 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271812    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:59.271938 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271833    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:59.271938 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.271859    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9301511d-3974-4f95-81c3-c41063623aeb-istiod-ca-cert" pod "9301511d-3974-4f95-81c3-c41063623aeb" (UID: "9301511d-3974-4f95-81c3-c41063623aeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:10:59.272261 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.272028    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-data" pod "9301511d-3974-4f95-81c3-c41063623aeb" (UID: "9301511d-3974-4f95-81c3-c41063623aeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:59.272261 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.272055    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-socket" pod "9301511d-3974-4f95-81c3-c41063623aeb" (UID: "9301511d-3974-4f95-81c3-c41063623aeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:59.274142 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.274102    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-kube-api-access-49hjc" pod "9301511d-3974-4f95-81c3-c41063623aeb" (UID: "9301511d-3974-4f95-81c3-c41063623aeb"). InnerVolumeSpecName "kube-api-access-49hjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:59.274518 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.274494    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-istio-token" pod "9301511d-3974-4f95-81c3-c41063623aeb" (UID: "9301511d-3974-4f95-81c3-c41063623aeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:10:59.274694 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.274666    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-envoy" pod "9301511d-3974-4f95-81c3-c41063623aeb" (UID: "9301511d-3974-4f95-81c3-c41063623aeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:10:59.274969 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.274934    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9301511d-3974-4f95-81c3-c41063623aeb-istio-podinfo" pod "9301511d-3974-4f95-81c3-c41063623aeb" (UID: "9301511d-3974-4f95-81c3-c41063623aeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:10:59.372507 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.372469    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:59.372507 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.372510    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9301511d-3974-4f95-81c3-c41063623aeb-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:59.372737 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.372524    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:59.372737 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.372538    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:59.372737 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.372551    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49hjc\" (UniqueName: \"kubernetes.io/projected/9301511d-3974-4f95-81c3-c41063623aeb-kube-api-access-49hjc\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:59.372737 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.372565    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9301511d-3974-4f95-81c3-c41063623aeb-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:59.372737 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.372580    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9301511d-3974-4f95-81c3-c41063623aeb-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:10:59.416441 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.416403    4494 generic.go:356] "Generic (PLEG): container finished" podID="9301511d-3974-4f95-81c3-c41063623aeb" containerID="ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d" exitCode=0
Oct 30 04:10:59.416638 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.416452    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" event={"ID":"9301511d-3974-4f95-81c3-c41063623aeb","Type":"ContainerDied","Data":"ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d"}
Oct 30 04:10:59.416638 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.416473    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"
Oct 30 04:10:59.416638 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.416498    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2" event={"ID":"9301511d-3974-4f95-81c3-c41063623aeb","Type":"ContainerDied","Data":"2d713c188a50b39a277e23cb01107fb4374848dc2b96d73e9baa4d84aa561d60"}
Oct 30 04:10:59.416638 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.416520    4494 scope.go:122] "RemoveContainer" containerID="ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d"
Oct 30 04:10:59.429955 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.429932    4494 scope.go:122] "RemoveContainer" containerID="0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578"
Oct 30 04:10:59.439365 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.439342    4494 scope.go:122] "RemoveContainer" containerID="ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d"
Oct 30 04:10:59.439775 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:10:59.439755    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d\": container with ID starting with ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d not found: ID does not exist" containerID="ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d"
Oct 30 04:10:59.439822 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.439789    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d"} err="failed to get container status \"ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d\": rpc error: code = NotFound desc = could not find container \"ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d\": container with ID starting with ff6d664668b0dfbcff10938bf5fd8a7273b31c320f792251330d23925182597d not found: ID does not exist"
Oct 30 04:10:59.439822 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.439810    4494 scope.go:122] "RemoveContainer" containerID="0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578"
Oct 30 04:10:59.440113 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:10:59.440094    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578\": container with ID starting with 0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578 not found: ID does not exist" containerID="0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578"
Oct 30 04:10:59.440180 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.440118    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578"} err="failed to get container status \"0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578\": rpc error: code = NotFound desc = could not find container \"0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578\": container with ID starting with 0db3709a92bd539e4957b8fee93f3876b94fb9f39d6934608ae627c4f6004578 not found: ID does not exist"
Oct 30 04:10:59.469728 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.469566    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"]
Oct 30 04:10:59.474060 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.474024    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-based-routing-only-f772d59b/model-routing-only-gateway-openshift-default-5f658554c7-b8wt2"]
Oct 30 04:10:59.927037 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:10:59.927005    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9301511d-3974-4f95-81c3-c41063623aeb" path="/var/lib/kubelet/pods/9301511d-3974-4f95-81c3-c41063623aeb/volumes"
Oct 30 04:11:24.083214 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:11:24.083162    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:11:54.083081 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:11:54.083042    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:12:13.176315 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.176279    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"]
Oct 30 04:12:13.176827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.176666    4494 cpu_manager.go:414] "containerMap: removing container" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy"
Oct 30 04:12:13.176827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.176680    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy"
Oct 30 04:12:13.176827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.176691    4494 cpu_manager.go:414] "containerMap: removing container" podUID="5b477960-9d27-4dbc-9bc9-add990d40f78" containerName="main"
Oct 30 04:12:13.176827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.176696    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b477960-9d27-4dbc-9bc9-add990d40f78" containerName="main"
Oct 30 04:12:13.176827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.176711    4494 cpu_manager.go:414] "containerMap: removing container" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy"
Oct 30 04:12:13.176827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.176716    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy"
Oct 30 04:12:13.176827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.176772    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy"
Oct 30 04:12:13.176827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.176785    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b477960-9d27-4dbc-9bc9-add990d40f78" containerName="main"
Oct 30 04:12:13.177151 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.176925    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="9301511d-3974-4f95-81c3-c41063623aeb" containerName="istio-proxy"
Oct 30 04:12:13.179742 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.179718    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.198548 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.198509    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Oct 30 04:12:13.198795 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.198742    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:12:13.199057 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.199036    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Oct 30 04:12:13.210858 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.210822    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-qshzj\""
Oct 30 04:12:13.211063 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.210872    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Oct 30 04:12:13.211063 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.210931    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-dockercfg-mgtnt\""
Oct 30 04:12:13.211063 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.210975    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Oct 30 04:12:13.213071 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.213045    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"]
Oct 30 04:12:13.290125 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.290079    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-tmp-dir\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.290321 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.290197    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk77l\" (UniqueName: \"kubernetes.io/projected/50816443-65a5-4317-9da2-7592ab3ef5f2-kube-api-access-pk77l\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.290321 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.290241    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-dshm\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.290321 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.290287    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-kserve-provision-location\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.290321 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.290314    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-home\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.290516 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.290412    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-model-cache\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.290516 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.290444    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50816443-65a5-4317-9da2-7592ab3ef5f2-tls-certs\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.290581 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.290537    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50816443-65a5-4317-9da2-7592ab3ef5f2-cabundle-cert\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.391736 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.391694    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50816443-65a5-4317-9da2-7592ab3ef5f2-cabundle-cert\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.391736 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.391742    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-tmp-dir\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392052 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.391777    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk77l\" (UniqueName: \"kubernetes.io/projected/50816443-65a5-4317-9da2-7592ab3ef5f2-kube-api-access-pk77l\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392052 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.391805    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-dshm\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392052 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.391823    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-kserve-provision-location\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392052 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.391844    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-home\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392052 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.391865    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-model-cache\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392052 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.391935    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50816443-65a5-4317-9da2-7592ab3ef5f2-tls-certs\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392350 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.392322    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-home\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.392393    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-kserve-provision-location\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392498 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.392475    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-tmp-dir\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392554 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.392523    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-model-cache\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.392605 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.392582    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50816443-65a5-4317-9da2-7592ab3ef5f2-cabundle-cert\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.394543 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.394513    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-dshm\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.394827 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.394811    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50816443-65a5-4317-9da2-7592ab3ef5f2-tls-certs\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.415389 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.415353    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk77l\" (UniqueName: \"kubernetes.io/projected/50816443-65a5-4317-9da2-7592ab3ef5f2-kube-api-access-pk77l\") pod \"tls-verification-test-kserve-5b966d4b74-w7vqs\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.493787 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.493690    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:13.815697 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.815656    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"]
Oct 30 04:12:13.825122 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:12:13.825088    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50816443_65a5_4317_9da2_7592ab3ef5f2.slice/crio-c04583c579af1b55c3686f7ea9f16d5722acce503601e5bbf8d8e707b96ffb84 WatchSource:0}: Error finding container c04583c579af1b55c3686f7ea9f16d5722acce503601e5bbf8d8e707b96ffb84: Status 404 returned error can't find the container with id c04583c579af1b55c3686f7ea9f16d5722acce503601e5bbf8d8e707b96ffb84
Oct 30 04:12:13.827127 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:13.827104    4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:12:14.799329 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:14.799289    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" event={"ID":"50816443-65a5-4317-9da2-7592ab3ef5f2","Type":"ContainerStarted","Data":"c04583c579af1b55c3686f7ea9f16d5722acce503601e5bbf8d8e707b96ffb84"}
Oct 30 04:12:15.804976 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:15.804933    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" event={"ID":"50816443-65a5-4317-9da2-7592ab3ef5f2","Type":"ContainerStarted","Data":"d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe"}
Oct 30 04:12:15.805488 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:15.805033    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:16.811724 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:16.811685    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" event={"ID":"50816443-65a5-4317-9da2-7592ab3ef5f2","Type":"ContainerStarted","Data":"ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900"}
Oct 30 04:12:18.823371 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:18.823335    4494 generic.go:356] "Generic (PLEG): container finished" podID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerID="ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900" exitCode=0
Oct 30 04:12:18.823809 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:18.823394    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" event={"ID":"50816443-65a5-4317-9da2-7592ab3ef5f2","Type":"ContainerDied","Data":"ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900"}
Oct 30 04:12:19.830208 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:19.830171    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" event={"ID":"50816443-65a5-4317-9da2-7592ab3ef5f2","Type":"ContainerStarted","Data":"7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260"}
Oct 30 04:12:19.903297 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:19.903238    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podStartSLOduration=3.166382102 podStartE2EDuration="6.903220288s" totalImagesPullingTime="1.568641532s" totalInitContainerRuntime="2.168196654s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:19.890750496 +0000 UTC m=+4604.096930715" watchObservedRunningTime="2026-10-30 04:12:19.903220288 +0000 UTC m=+4604.109400542"
Oct 30 04:12:23.494149 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:23.494101    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:23.494627 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:23.494184    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:23.495660 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:23.495628    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:12:24.084927 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:12:24.084870    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:12:33.494856 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:33.494792    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:12:33.507934 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:33.507877    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:12:36.481666 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:36.481614    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 04:12:43.494405 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:43.494335    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:12:51.507234 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:51.507102    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="5988729d-daeb-4532-9716-9a464840c988" containerName="main" containerID="cri-o://e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2" gracePeriod=45
Oct 30 04:12:52.255510 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.255480    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:12:52.391892 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.391755    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-dshm\") pod \"5988729d-daeb-4532-9716-9a464840c988\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") "
Oct 30 04:12:52.391892 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.391855    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-tmp-dir\") pod \"5988729d-daeb-4532-9716-9a464840c988\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") "
Oct 30 04:12:52.391892 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.391880    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5988729d-daeb-4532-9716-9a464840c988-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5988729d-daeb-4532-9716-9a464840c988-cabundle-cert\") pod \"5988729d-daeb-4532-9716-9a464840c988\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") "
Oct 30 04:12:52.392228 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.391923    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-model-cache\") pod \"5988729d-daeb-4532-9716-9a464840c988\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") "
Oct 30 04:12:52.392228 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.392052    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5988729d-daeb-4532-9716-9a464840c988-tls-certs\" (UniqueName: \"kubernetes.io/secret/5988729d-daeb-4532-9716-9a464840c988-tls-certs\") pod \"5988729d-daeb-4532-9716-9a464840c988\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") "
Oct 30 04:12:52.392228 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.392120    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-home\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-home\") pod \"5988729d-daeb-4532-9716-9a464840c988\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") "
Oct 30 04:12:52.392228 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.392175    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5988729d-daeb-4532-9716-9a464840c988-kube-api-access-8k4nf\" (UniqueName: \"kubernetes.io/projected/5988729d-daeb-4532-9716-9a464840c988-kube-api-access-8k4nf\") pod \"5988729d-daeb-4532-9716-9a464840c988\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") "
Oct 30 04:12:52.392228 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.392185    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-model-cache" pod "5988729d-daeb-4532-9716-9a464840c988" (UID: "5988729d-daeb-4532-9716-9a464840c988"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:52.392228 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.392224    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-kserve-provision-location\") pod \"5988729d-daeb-4532-9716-9a464840c988\" (UID: \"5988729d-daeb-4532-9716-9a464840c988\") "
Oct 30 04:12:52.392554 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.392287    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5988729d-daeb-4532-9716-9a464840c988-cabundle-cert" pod "5988729d-daeb-4532-9716-9a464840c988" (UID: "5988729d-daeb-4532-9716-9a464840c988"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:12:52.392611 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.392567    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5988729d-daeb-4532-9716-9a464840c988-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.392611 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.392588    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.392944 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.392884    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-home" pod "5988729d-daeb-4532-9716-9a464840c988" (UID: "5988729d-daeb-4532-9716-9a464840c988"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:52.395333 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.395298    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-dshm" pod "5988729d-daeb-4532-9716-9a464840c988" (UID: "5988729d-daeb-4532-9716-9a464840c988"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:52.395464 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.395354    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5988729d-daeb-4532-9716-9a464840c988-tls-certs" pod "5988729d-daeb-4532-9716-9a464840c988" (UID: "5988729d-daeb-4532-9716-9a464840c988"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:12:52.395464 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.395360    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5988729d-daeb-4532-9716-9a464840c988-kube-api-access-8k4nf" pod "5988729d-daeb-4532-9716-9a464840c988" (UID: "5988729d-daeb-4532-9716-9a464840c988"). InnerVolumeSpecName "kube-api-access-8k4nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:12:52.409850 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.409799    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-tmp-dir" pod "5988729d-daeb-4532-9716-9a464840c988" (UID: "5988729d-daeb-4532-9716-9a464840c988"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:52.418732 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.418679    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-kserve-provision-location" pod "5988729d-daeb-4532-9716-9a464840c988" (UID: "5988729d-daeb-4532-9716-9a464840c988"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:12:52.493117 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.493074    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.493117 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.493106    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.493117 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.493115    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5988729d-daeb-4532-9716-9a464840c988-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.493117 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.493122    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.493117 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.493131    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k4nf\" (UniqueName: \"kubernetes.io/projected/5988729d-daeb-4532-9716-9a464840c988-kube-api-access-8k4nf\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:12:52.493554 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:52.493140    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5988729d-daeb-4532-9716-9a464840c988-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:12:53.020012 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.019968    4494 generic.go:356] "Generic (PLEG): container finished" podID="5988729d-daeb-4532-9716-9a464840c988" containerID="e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2" exitCode=0
Oct 30 04:12:53.020570 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.020035    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5988729d-daeb-4532-9716-9a464840c988","Type":"ContainerDied","Data":"e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2"}
Oct 30 04:12:53.020570 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.020042    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 04:12:53.020570 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.020093    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"5988729d-daeb-4532-9716-9a464840c988","Type":"ContainerDied","Data":"5f3a602bd4eae7099743766e0fb17224c4fedad633f371850feb38902ea4f38a"}
Oct 30 04:12:53.020570 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.020128    4494 scope.go:122] "RemoveContainer" containerID="e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2"
Oct 30 04:12:53.030646 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.030622    4494 scope.go:122] "RemoveContainer" containerID="efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4"
Oct 30 04:12:53.042232 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.042199    4494 scope.go:122] "RemoveContainer" containerID="e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2"
Oct 30 04:12:53.042629 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:12:53.042593    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2\": container with ID starting with e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2 not found: ID does not exist" containerID="e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2"
Oct 30 04:12:53.042722 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.042640    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2"} err="failed to get container status \"e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2\": rpc error: code = NotFound desc = could not find container \"e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2\": container with ID starting with e701f4ec4449982c4e2ba924d888bb7d81c4cae54fa3d92266df116928fccda2 not found: ID does not exist"
Oct 30 04:12:53.042722 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.042661    4494 scope.go:122] "RemoveContainer" containerID="efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4"
Oct 30 04:12:53.043010 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:12:53.042991    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4\": container with ID starting with efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4 not found: ID does not exist" containerID="efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4"
Oct 30 04:12:53.043088 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.043020    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4"} err="failed to get container status \"efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4\": rpc error: code = NotFound desc = could not find container \"efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4\": container with ID starting with efd8bf8fdf54b1dac13364bdfe59a3f61d048add9c9014acf9a60355083330c4 not found: ID does not exist"
Oct 30 04:12:53.084181 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.084143    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 04:12:53.089968 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.089918    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 04:12:53.494801 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.494747    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:12:53.926978 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:12:53.926942    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5988729d-daeb-4532-9716-9a464840c988" path="/var/lib/kubelet/pods/5988729d-daeb-4532-9716-9a464840c988/volumes"
Oct 30 04:12:54.085721 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:12:54.085677    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:13:03.495227 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:13:03.495171    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:13:13.494487 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:13:13.494438    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:13:23.495138 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:13:23.495082    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:13:24.083475 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:13:24.083434    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:13:33.495201 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:13:33.495150    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:13:36.839306 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:13:36.839258    4494 scope.go:122] "RemoveContainer" containerID="ab0e007ebcccb657d66d21f7e62d1136434766193b8ec660950c64d1ac6c8636"
Oct 30 04:13:43.494416 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:13:43.494367    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:13:53.494191 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:13:53.494129    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:13:54.084882 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:13:54.084841    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:14:03.494838 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:14:03.494785    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:14:13.494499 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:14:13.494432    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8001/health\": dial tcp 10.134.0.92:8001: connect: connection refused"
Oct 30 04:14:23.510617 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:14:23.510576    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:14:23.523064 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:14:23.523026    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:14:24.084411 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:14:24.084372    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:14:54.082780 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:14:54.082737    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:15:16.631015 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.630980    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"]
Oct 30 04:15:16.631482 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.631394    4494 cpu_manager.go:414] "containerMap: removing container" podUID="5988729d-daeb-4532-9716-9a464840c988" containerName="main"
Oct 30 04:15:16.631482 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.631408    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5988729d-daeb-4532-9716-9a464840c988" containerName="main"
Oct 30 04:15:16.631482 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.631422    4494 cpu_manager.go:414] "containerMap: removing container" podUID="5988729d-daeb-4532-9716-9a464840c988" containerName="storage-initializer"
Oct 30 04:15:16.631482 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.631429    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5988729d-daeb-4532-9716-9a464840c988" containerName="storage-initializer"
Oct 30 04:15:16.631699 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.631541    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="5988729d-daeb-4532-9716-9a464840c988" containerName="main"
Oct 30 04:15:16.635067 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.635050    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.649331 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.649305    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"]
Oct 30 04:15:16.659771 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.659732    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Oct 30 04:15:16.659771 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.659737    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Oct 30 04:15:16.659771 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.659736    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:15:16.659771 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.659772    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Oct 30 04:15:16.660058 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.659737    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-vcvsf\""
Oct 30 04:15:16.660058 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.659739    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Oct 30 04:15:16.697005 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.696963    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tls-certs\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.697005 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.697008    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rz7\" (UniqueName: \"kubernetes.io/projected/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kube-api-access-f5rz7\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.697296 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.697185    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-dshm\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.697296 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.697243    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tmp-dir\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.697430 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.697340    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-model-cache\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.697430 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.697373    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kserve-provision-location\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.697430 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.697409    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-home\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.697589 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.697569    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-cabundle-cert\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.798825 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.798774    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tls-certs\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799082 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.798833    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rz7\" (UniqueName: \"kubernetes.io/projected/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kube-api-access-f5rz7\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799082 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.798867    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-dshm\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799082 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.798932    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tmp-dir\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799082 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.798973    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-model-cache\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799082 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.798996    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kserve-provision-location\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799082 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.799023    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-home\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799426 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.799265    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-cabundle-cert\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799484 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.799458    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kserve-provision-location\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799591 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.799564    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-model-cache\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799659 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.799577    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tmp-dir\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799826 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.799804    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-cabundle-cert\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.799887 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.799836    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-home\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.801519 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.801496    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-dshm\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.801719 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.801700    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tls-certs\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.823153 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.823122    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rz7\" (UniqueName: \"kubernetes.io/projected/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kube-api-access-f5rz7\") pod \"prestop-hook-test-kserve-564c47989f-c2qf2\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:16.951522 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:16.951415    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:17.273172 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:17.273132    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"]
Oct 30 04:15:17.273862 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:15:17.273827    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168bf7fe_7e7c_4cfe_b73e_e0d90960353a.slice/crio-a56b6a5bc058f5e55ade8a88940de89d21c5e225c2ab9c7498ec429b485e056f WatchSource:0}: Error finding container a56b6a5bc058f5e55ade8a88940de89d21c5e225c2ab9c7498ec429b485e056f: Status 404 returned error can't find the container with id a56b6a5bc058f5e55ade8a88940de89d21c5e225c2ab9c7498ec429b485e056f
Oct 30 04:15:17.718973 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:17.718930    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2" event={"ID":"168bf7fe-7e7c-4cfe-b73e-e0d90960353a","Type":"ContainerStarted","Data":"46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4"}
Oct 30 04:15:17.719465 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:17.718996    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2" event={"ID":"168bf7fe-7e7c-4cfe-b73e-e0d90960353a","Type":"ContainerStarted","Data":"a56b6a5bc058f5e55ade8a88940de89d21c5e225c2ab9c7498ec429b485e056f"}
Oct 30 04:15:19.730708 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:19.730668    4494 generic.go:356] "Generic (PLEG): container finished" podID="168bf7fe-7e7c-4cfe-b73e-e0d90960353a" containerID="46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4" exitCode=0
Oct 30 04:15:19.731250 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:19.730735    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2" event={"ID":"168bf7fe-7e7c-4cfe-b73e-e0d90960353a","Type":"ContainerDied","Data":"46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4"}
Oct 30 04:15:20.739229 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:20.739188    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2" event={"ID":"168bf7fe-7e7c-4cfe-b73e-e0d90960353a","Type":"ContainerStarted","Data":"2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c"}
Oct 30 04:15:20.812100 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:20.811994    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2" podStartSLOduration=2.476370893 podStartE2EDuration="4.811971526s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.335600633s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:20.798246241 +0000 UTC m=+4785.004426482" watchObservedRunningTime="2026-10-30 04:15:20.811971526 +0000 UTC m=+4785.018151738"
Oct 30 04:15:20.951770 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:20.951729    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:24.084949 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:15:24.084914    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:15:26.952192 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:26.952141    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:15:26.954051 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:26.954020    4494 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2" podUID="168bf7fe-7e7c-4cfe-b73e-e0d90960353a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": dial tcp 10.134.0.93:8000: connect: connection refused"
Oct 30 04:15:35.378814 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:35.378773    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"]
Oct 30 04:15:50.404050 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:15:50.403946    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2" podUID="168bf7fe-7e7c-4cfe-b73e-e0d90960353a" containerName="main" containerID="cri-o://2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c" gracePeriod=45
Oct 30 04:15:54.085405 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:15:54.085354    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:16:24.083891 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:16:24.083846    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:16:36.083234 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.083192    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-564c47989f-c2qf2_168bf7fe-7e7c-4cfe-b73e-e0d90960353a/main/0.log"
Oct 30 04:16:36.087391 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.087350    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:16:36.120234 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.120193    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-564c47989f-c2qf2_168bf7fe-7e7c-4cfe-b73e-e0d90960353a/main/0.log"
Oct 30 04:16:36.121033 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.120974    4494 generic.go:356] "Generic (PLEG): container finished" podID="168bf7fe-7e7c-4cfe-b73e-e0d90960353a" containerID="2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c" exitCode=137
Oct 30 04:16:36.121221 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.121193    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2" event={"ID":"168bf7fe-7e7c-4cfe-b73e-e0d90960353a","Type":"ContainerDied","Data":"2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c"}
Oct 30 04:16:36.121313 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.121286    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2" event={"ID":"168bf7fe-7e7c-4cfe-b73e-e0d90960353a","Type":"ContainerDied","Data":"a56b6a5bc058f5e55ade8a88940de89d21c5e225c2ab9c7498ec429b485e056f"}
Oct 30 04:16:36.121377 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.121341    4494 scope.go:122] "RemoveContainer" containerID="2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c"
Oct 30 04:16:36.121617 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.121595    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"
Oct 30 04:16:36.131932 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.131910    4494 scope.go:122] "RemoveContainer" containerID="46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4"
Oct 30 04:16:36.140300 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.140272    4494 scope.go:122] "RemoveContainer" containerID="2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c"
Oct 30 04:16:36.140601 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:16:36.140579    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c\": container with ID starting with 2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c not found: ID does not exist" containerID="2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c"
Oct 30 04:16:36.140662 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.140611    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c"} err="failed to get container status \"2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c\": rpc error: code = NotFound desc = could not find container \"2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c\": container with ID starting with 2e640703939cf79baa6fb3ba58edbcd4be5082522e744e6c058500466d9b873c not found: ID does not exist"
Oct 30 04:16:36.140662 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.140631    4494 scope.go:122] "RemoveContainer" containerID="46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4"
Oct 30 04:16:36.140942 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:16:36.140927    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4\": container with ID starting with 46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4 not found: ID does not exist" containerID="46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4"
Oct 30 04:16:36.141012 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.140945    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4"} err="failed to get container status \"46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4\": rpc error: code = NotFound desc = could not find container \"46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4\": container with ID starting with 46f21e5cb9d8bca055f1f404b3eac6713b8d3204d1d3627e449612fa43d8b7f4 not found: ID does not exist"
Oct 30 04:16:36.203930 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.203808    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-home\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-home\") pod \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") "
Oct 30 04:16:36.203930 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.203916    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-dshm\") pod \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") "
Oct 30 04:16:36.204168 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.203955    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kserve-provision-location\") pod \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") "
Oct 30 04:16:36.204168 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.204010    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kube-api-access-f5rz7\" (UniqueName: \"kubernetes.io/projected/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kube-api-access-f5rz7\") pod \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") "
Oct 30 04:16:36.204168 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.204073    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-cabundle-cert\") pod \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") "
Oct 30 04:16:36.204168 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.204109    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tls-certs\" (UniqueName: \"kubernetes.io/secret/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tls-certs\") pod \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") "
Oct 30 04:16:36.204168 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.204136    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tmp-dir\") pod \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") "
Oct 30 04:16:36.204168 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.204160    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-model-cache\") pod \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\" (UID: \"168bf7fe-7e7c-4cfe-b73e-e0d90960353a\") "
Oct 30 04:16:36.204726 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.204664    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-model-cache" pod "168bf7fe-7e7c-4cfe-b73e-e0d90960353a" (UID: "168bf7fe-7e7c-4cfe-b73e-e0d90960353a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:36.205217 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.205174    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-home" pod "168bf7fe-7e7c-4cfe-b73e-e0d90960353a" (UID: "168bf7fe-7e7c-4cfe-b73e-e0d90960353a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:36.205381 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.205262    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tmp-dir" pod "168bf7fe-7e7c-4cfe-b73e-e0d90960353a" (UID: "168bf7fe-7e7c-4cfe-b73e-e0d90960353a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:36.205611 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.205582    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-cabundle-cert" pod "168bf7fe-7e7c-4cfe-b73e-e0d90960353a" (UID: "168bf7fe-7e7c-4cfe-b73e-e0d90960353a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:36.212227 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.212182    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tls-certs" pod "168bf7fe-7e7c-4cfe-b73e-e0d90960353a" (UID: "168bf7fe-7e7c-4cfe-b73e-e0d90960353a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:36.222584 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.222541    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kube-api-access-f5rz7" pod "168bf7fe-7e7c-4cfe-b73e-e0d90960353a" (UID: "168bf7fe-7e7c-4cfe-b73e-e0d90960353a"). InnerVolumeSpecName "kube-api-access-f5rz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:36.222867 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.222838    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-dshm" pod "168bf7fe-7e7c-4cfe-b73e-e0d90960353a" (UID: "168bf7fe-7e7c-4cfe-b73e-e0d90960353a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:36.237483 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.237426    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kserve-provision-location" pod "168bf7fe-7e7c-4cfe-b73e-e0d90960353a" (UID: "168bf7fe-7e7c-4cfe-b73e-e0d90960353a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:36.305242 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.305200    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:36.305242 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.305238    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:36.305242 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.305247    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:36.305501 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.305256    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:36.305501 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.305264    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:36.305501 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.305271    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:36.305501 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.305279    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:36.305501 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.305288    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5rz7\" (UniqueName: \"kubernetes.io/projected/168bf7fe-7e7c-4cfe-b73e-e0d90960353a-kube-api-access-f5rz7\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:36.469429 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.469318    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"]
Oct 30 04:16:36.482332 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:36.482296    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-564c47989f-c2qf2"]
Oct 30 04:16:37.682267 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:37.682225    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"]
Oct 30 04:16:37.927582 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:37.927541    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168bf7fe-7e7c-4cfe-b73e-e0d90960353a" path="/var/lib/kubelet/pods/168bf7fe-7e7c-4cfe-b73e-e0d90960353a/volumes"
Oct 30 04:16:52.707237 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:52.707167    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" containerID="cri-o://7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260" gracePeriod=45
Oct 30 04:16:53.325091 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.325039    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="llm-d-routing-sidecar" containerID="cri-o://d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe" gracePeriod=45
Oct 30 04:16:53.494667 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.494622    4494 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": dial tcp 10.134.0.92:8000: connect: connection refused"
Oct 30 04:16:53.494848 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.494750    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": dial tcp 10.134.0.92:8000: connect: connection refused"
Oct 30 04:16:53.583188 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.583159    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:16:53.660195 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.660149    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-dshm\") pod \"50816443-65a5-4317-9da2-7592ab3ef5f2\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") "
Oct 30 04:16:53.660390 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.660205    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50816443-65a5-4317-9da2-7592ab3ef5f2-kube-api-access-pk77l\" (UniqueName: \"kubernetes.io/projected/50816443-65a5-4317-9da2-7592ab3ef5f2-kube-api-access-pk77l\") pod \"50816443-65a5-4317-9da2-7592ab3ef5f2\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") "
Oct 30 04:16:53.660390 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.660245    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50816443-65a5-4317-9da2-7592ab3ef5f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/50816443-65a5-4317-9da2-7592ab3ef5f2-tls-certs\") pod \"50816443-65a5-4317-9da2-7592ab3ef5f2\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") "
Oct 30 04:16:53.660390 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.660303    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-tmp-dir\") pod \"50816443-65a5-4317-9da2-7592ab3ef5f2\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") "
Oct 30 04:16:53.660390 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.660356    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50816443-65a5-4317-9da2-7592ab3ef5f2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50816443-65a5-4317-9da2-7592ab3ef5f2-cabundle-cert\") pod \"50816443-65a5-4317-9da2-7592ab3ef5f2\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") "
Oct 30 04:16:53.660635 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.660437    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-model-cache\") pod \"50816443-65a5-4317-9da2-7592ab3ef5f2\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") "
Oct 30 04:16:53.660635 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.660482    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-kserve-provision-location\") pod \"50816443-65a5-4317-9da2-7592ab3ef5f2\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") "
Oct 30 04:16:53.660635 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.660505    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-home\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-home\") pod \"50816443-65a5-4317-9da2-7592ab3ef5f2\" (UID: \"50816443-65a5-4317-9da2-7592ab3ef5f2\") "
Oct 30 04:16:53.661560 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.661490    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-home" pod "50816443-65a5-4317-9da2-7592ab3ef5f2" (UID: "50816443-65a5-4317-9da2-7592ab3ef5f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.663395 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.663055    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50816443-65a5-4317-9da2-7592ab3ef5f2-cabundle-cert" pod "50816443-65a5-4317-9da2-7592ab3ef5f2" (UID: "50816443-65a5-4317-9da2-7592ab3ef5f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:16:53.663395 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.663262    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-dshm" pod "50816443-65a5-4317-9da2-7592ab3ef5f2" (UID: "50816443-65a5-4317-9da2-7592ab3ef5f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.663395 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.663286    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50816443-65a5-4317-9da2-7592ab3ef5f2-tls-certs" pod "50816443-65a5-4317-9da2-7592ab3ef5f2" (UID: "50816443-65a5-4317-9da2-7592ab3ef5f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:16:53.663395 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.663302    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50816443-65a5-4317-9da2-7592ab3ef5f2-kube-api-access-pk77l" pod "50816443-65a5-4317-9da2-7592ab3ef5f2" (UID: "50816443-65a5-4317-9da2-7592ab3ef5f2"). InnerVolumeSpecName "kube-api-access-pk77l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:16:53.663395 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.663361    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-model-cache" pod "50816443-65a5-4317-9da2-7592ab3ef5f2" (UID: "50816443-65a5-4317-9da2-7592ab3ef5f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.674175 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.674145    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-tmp-dir" pod "50816443-65a5-4317-9da2-7592ab3ef5f2" (UID: "50816443-65a5-4317-9da2-7592ab3ef5f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.681764 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.681736    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-kserve-provision-location" pod "50816443-65a5-4317-9da2-7592ab3ef5f2" (UID: "50816443-65a5-4317-9da2-7592ab3ef5f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:16:53.761512 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.761467    4494 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-kserve-provision-location\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.761512 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.761505    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.761512 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.761515    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.761512 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.761524    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk77l\" (UniqueName: \"kubernetes.io/projected/50816443-65a5-4317-9da2-7592ab3ef5f2-kube-api-access-pk77l\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.762098 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.761533    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50816443-65a5-4317-9da2-7592ab3ef5f2-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.762098 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.761563    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.762098 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.761571    4494 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50816443-65a5-4317-9da2-7592ab3ef5f2-cabundle-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:53.762098 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:53.761579    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50816443-65a5-4317-9da2-7592ab3ef5f2-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:16:54.083183 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:16:54.083139    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:16:54.202484 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.202376    4494 generic.go:356] "Generic (PLEG): container finished" podID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerID="7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260" exitCode=0
Oct 30 04:16:54.202484 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.202402    4494 generic.go:356] "Generic (PLEG): container finished" podID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerID="d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe" exitCode=0
Oct 30 04:16:54.202484 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.202438    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"
Oct 30 04:16:54.202484 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.202441    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" event={"ID":"50816443-65a5-4317-9da2-7592ab3ef5f2","Type":"ContainerDied","Data":"7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260"}
Oct 30 04:16:54.202835 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.202516    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" event={"ID":"50816443-65a5-4317-9da2-7592ab3ef5f2","Type":"ContainerDied","Data":"d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe"}
Oct 30 04:16:54.202835 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.202541    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" event={"ID":"50816443-65a5-4317-9da2-7592ab3ef5f2","Type":"ContainerDied","Data":"c04583c579af1b55c3686f7ea9f16d5722acce503601e5bbf8d8e707b96ffb84"}
Oct 30 04:16:54.202835 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.202568    4494 scope.go:122] "RemoveContainer" containerID="7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260"
Oct 30 04:16:54.211959 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.211934    4494 scope.go:122] "RemoveContainer" containerID="ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900"
Oct 30 04:16:54.220517 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.220496    4494 scope.go:122] "RemoveContainer" containerID="d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe"
Oct 30 04:16:54.228800 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.228777    4494 scope.go:122] "RemoveContainer" containerID="7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260"
Oct 30 04:16:54.229144 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:16:54.229123    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260\": container with ID starting with 7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260 not found: ID does not exist" containerID="7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260"
Oct 30 04:16:54.229221 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.229159    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260"} err="failed to get container status \"7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260\": rpc error: code = NotFound desc = could not find container \"7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260\": container with ID starting with 7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260 not found: ID does not exist"
Oct 30 04:16:54.229221 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.229185    4494 scope.go:122] "RemoveContainer" containerID="ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900"
Oct 30 04:16:54.229462 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:16:54.229439    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900\": container with ID starting with ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900 not found: ID does not exist" containerID="ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900"
Oct 30 04:16:54.229502 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.229476    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900"} err="failed to get container status \"ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900\": rpc error: code = NotFound desc = could not find container \"ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900\": container with ID starting with ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900 not found: ID does not exist"
Oct 30 04:16:54.229550 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.229510    4494 scope.go:122] "RemoveContainer" containerID="d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe"
Oct 30 04:16:54.229762 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:16:54.229742    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe\": container with ID starting with d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe not found: ID does not exist" containerID="d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe"
Oct 30 04:16:54.229830 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.229767    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe"} err="failed to get container status \"d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe\": rpc error: code = NotFound desc = could not find container \"d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe\": container with ID starting with d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe not found: ID does not exist"
Oct 30 04:16:54.229830 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.229782    4494 scope.go:122] "RemoveContainer" containerID="7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260"
Oct 30 04:16:54.230324 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.230297    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260"} err="failed to get container status \"7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260\": rpc error: code = NotFound desc = could not find container \"7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260\": container with ID starting with 7627a31119d72019930820e425d1bb8b64c7a600f291da2c85fc08be3c467260 not found: ID does not exist"
Oct 30 04:16:54.230324 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.230322    4494 scope.go:122] "RemoveContainer" containerID="ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900"
Oct 30 04:16:54.230614 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.230592    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900"} err="failed to get container status \"ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900\": rpc error: code = NotFound desc = could not find container \"ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900\": container with ID starting with ddb6aed795a5a77c5a8ec59413055dfd9870a5d742af8be89521a2ea804df900 not found: ID does not exist"
Oct 30 04:16:54.230698 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.230616    4494 scope.go:122] "RemoveContainer" containerID="d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe"
Oct 30 04:16:54.230912 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.230877    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe"} err="failed to get container status \"d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe\": rpc error: code = NotFound desc = could not find container \"d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe\": container with ID starting with d96691a90dcb29b68eccffd59676e339807e9b1ec1e8f5af01ca274011bca4fe not found: ID does not exist"
Oct 30 04:16:54.268832 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.268783    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"]
Oct 30 04:16:54.281280 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:54.281245    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs"]
Oct 30 04:16:55.926435 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:55.926399    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" path="/var/lib/kubelet/pods/50816443-65a5-4317-9da2-7592ab3ef5f2/volumes"
Oct 30 04:16:58.512311 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:16:58.512258    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-5b966d4b74-w7vqs" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 04:17:24.081496 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:17:24.081460    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:17:38.929926 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:38.929809    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"]
Oct 30 04:17:38.930533 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:38.930134    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerName="istio-proxy" containerID="cri-o://a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7" gracePeriod=30
Oct 30 04:17:38.958942 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:38.958340    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"]
Oct 30 04:17:38.958942 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:38.958577    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerName="istio-proxy" containerID="cri-o://9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39" gracePeriod=30
Oct 30 04:17:38.960843 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:38.960808    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"]
Oct 30 04:17:38.961084 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:38.961056    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" containerName="istio-proxy" containerID="cri-o://4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45" gracePeriod=30
Oct 30 04:17:43.895224 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:43.895183    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:17:43.895821 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:43.895248    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:17:43.932534 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:43.932500    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:17:43.932717 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:43.932559    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:17:44.357379 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:44.357338    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 04:17:44.357582 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:44.357412    4494 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 04:17:48.868377 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.868347    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:17:48.871370 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.871344    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:17:48.883628 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.883605    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:17:48.968063 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968031    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-data\") pod \"33f46c43-2286-48da-bb3c-acf0bd23720a\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") "
Oct 30 04:17:48.968256 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968081    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-credential-socket\") pod \"c4b08259-c22f-4be6-a144-4049edf81dca\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") "
Oct 30 04:17:48.968256 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968130    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-data\") pod \"c4b08259-c22f-4be6-a144-4049edf81dca\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") "
Oct 30 04:17:48.968256 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968169    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-socket\") pod \"33f46c43-2286-48da-bb3c-acf0bd23720a\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") "
Oct 30 04:17:48.968256 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968195    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-socket\") pod \"c4b08259-c22f-4be6-a144-4049edf81dca\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") "
Oct 30 04:17:48.968256 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968241    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4b08259-c22f-4be6-a144-4049edf81dca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b08259-c22f-4be6-a144-4049edf81dca-istio-podinfo\") pod \"c4b08259-c22f-4be6-a144-4049edf81dca\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") "
Oct 30 04:17:48.968535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968272    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-kube-api-access-hbslj\" (UniqueName: \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-kube-api-access-hbslj\") pod \"c4b08259-c22f-4be6-a144-4049edf81dca\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") "
Oct 30 04:17:48.968535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968307    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b08259-c22f-4be6-a144-4049edf81dca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b08259-c22f-4be6-a144-4049edf81dca-istiod-ca-cert\") pod \"c4b08259-c22f-4be6-a144-4049edf81dca\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") "
Oct 30 04:17:48.968535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968328    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-credential-socket" pod "c4b08259-c22f-4be6-a144-4049edf81dca" (UID: "c4b08259-c22f-4be6-a144-4049edf81dca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:48.968535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968356    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-certs\") pod \"33f46c43-2286-48da-bb3c-acf0bd23720a\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") "
Oct 30 04:17:48.968535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968387    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-podinfo\") pod \"33f46c43-2286-48da-bb3c-acf0bd23720a\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") "
Oct 30 04:17:48.968535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968354    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-data" pod "33f46c43-2286-48da-bb3c-acf0bd23720a" (UID: "33f46c43-2286-48da-bb3c-acf0bd23720a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:48.968535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968443    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-data" pod "c4b08259-c22f-4be6-a144-4049edf81dca" (UID: "c4b08259-c22f-4be6-a144-4049edf81dca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:48.968535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968457    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-kube-api-access-htlw5\" (UniqueName: \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-kube-api-access-htlw5\") pod \"33f46c43-2286-48da-bb3c-acf0bd23720a\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") "
Oct 30 04:17:48.968535 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968525    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-envoy\") pod \"c4b08259-c22f-4be6-a144-4049edf81dca\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") "
Oct 30 04:17:48.969032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968607    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f46c43-2286-48da-bb3c-acf0bd23720a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f46c43-2286-48da-bb3c-acf0bd23720a-istiod-ca-cert\") pod \"33f46c43-2286-48da-bb3c-acf0bd23720a\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") "
Oct 30 04:17:48.969032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968619    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-socket" pod "33f46c43-2286-48da-bb3c-acf0bd23720a" (UID: "33f46c43-2286-48da-bb3c-acf0bd23720a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:48.969032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968643    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-credential-socket\") pod \"33f46c43-2286-48da-bb3c-acf0bd23720a\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") "
Oct 30 04:17:48.969032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968675    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-certs\") pod \"c4b08259-c22f-4be6-a144-4049edf81dca\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") "
Oct 30 04:17:48.969032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968703    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-istio-token\" (UniqueName: \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-istio-token\") pod \"c4b08259-c22f-4be6-a144-4049edf81dca\" (UID: \"c4b08259-c22f-4be6-a144-4049edf81dca\") "
Oct 30 04:17:48.969032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968748    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-envoy\") pod \"33f46c43-2286-48da-bb3c-acf0bd23720a\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") "
Oct 30 04:17:48.969032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968774    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-token\" (UniqueName: \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-token\") pod \"33f46c43-2286-48da-bb3c-acf0bd23720a\" (UID: \"33f46c43-2286-48da-bb3c-acf0bd23720a\") "
Oct 30 04:17:48.969032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968980    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-socket" pod "c4b08259-c22f-4be6-a144-4049edf81dca" (UID: "c4b08259-c22f-4be6-a144-4049edf81dca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:48.969032 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.968987    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-certs" pod "33f46c43-2286-48da-bb3c-acf0bd23720a" (UID: "33f46c43-2286-48da-bb3c-acf0bd23720a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:48.969659 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.969340    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:48.969659 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.969362    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:48.969659 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.969366    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-certs" pod "c4b08259-c22f-4be6-a144-4049edf81dca" (UID: "c4b08259-c22f-4be6-a144-4049edf81dca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:48.969659 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.969375    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:48.969659 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.969389    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:48.969659 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.969403    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:48.969659 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.969416    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:48.970066 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.969759    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f46c43-2286-48da-bb3c-acf0bd23720a-istiod-ca-cert" pod "33f46c43-2286-48da-bb3c-acf0bd23720a" (UID: "33f46c43-2286-48da-bb3c-acf0bd23720a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:48.970066 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.970056    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-credential-socket" pod "33f46c43-2286-48da-bb3c-acf0bd23720a" (UID: "33f46c43-2286-48da-bb3c-acf0bd23720a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:48.970497 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.970469    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b08259-c22f-4be6-a144-4049edf81dca-istiod-ca-cert" pod "c4b08259-c22f-4be6-a144-4049edf81dca" (UID: "c4b08259-c22f-4be6-a144-4049edf81dca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:48.972095 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.972065    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-podinfo" pod "33f46c43-2286-48da-bb3c-acf0bd23720a" (UID: "33f46c43-2286-48da-bb3c-acf0bd23720a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:17:48.972095 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.972092    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-kube-api-access-htlw5" pod "33f46c43-2286-48da-bb3c-acf0bd23720a" (UID: "33f46c43-2286-48da-bb3c-acf0bd23720a"). InnerVolumeSpecName "kube-api-access-htlw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:48.972095 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.972105    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-token" pod "33f46c43-2286-48da-bb3c-acf0bd23720a" (UID: "33f46c43-2286-48da-bb3c-acf0bd23720a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:48.972380 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.972159    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-kube-api-access-hbslj" pod "c4b08259-c22f-4be6-a144-4049edf81dca" (UID: "c4b08259-c22f-4be6-a144-4049edf81dca"). InnerVolumeSpecName "kube-api-access-hbslj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:48.972380 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.972260    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4b08259-c22f-4be6-a144-4049edf81dca-istio-podinfo" pod "c4b08259-c22f-4be6-a144-4049edf81dca" (UID: "c4b08259-c22f-4be6-a144-4049edf81dca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:17:48.972674 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.972656    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-istio-token" pod "c4b08259-c22f-4be6-a144-4049edf81dca" (UID: "c4b08259-c22f-4be6-a144-4049edf81dca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:48.972961 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.972884    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-envoy" pod "33f46c43-2286-48da-bb3c-acf0bd23720a" (UID: "33f46c43-2286-48da-bb3c-acf0bd23720a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:48.973512 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:48.973494    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-envoy" pod "c4b08259-c22f-4be6-a144-4049edf81dca" (UID: "c4b08259-c22f-4be6-a144-4049edf81dca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:49.070027 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.069982    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-credential-socket\") pod \"d85914c5-e17a-4115-afe1-b0815bcdad36\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") "
Oct 30 04:17:49.070250 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070082    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-token\" (UniqueName: \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-token\") pod \"d85914c5-e17a-4115-afe1-b0815bcdad36\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") "
Oct 30 04:17:49.070250 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070128    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-podinfo\") pod \"d85914c5-e17a-4115-afe1-b0815bcdad36\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") "
Oct 30 04:17:49.070250 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070159    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-kube-api-access-cpn8z\" (UniqueName: \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-kube-api-access-cpn8z\") pod \"d85914c5-e17a-4115-afe1-b0815bcdad36\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") "
Oct 30 04:17:49.070250 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070191    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-socket\") pod \"d85914c5-e17a-4115-afe1-b0815bcdad36\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") "
Oct 30 04:17:49.070250 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070221    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-envoy\") pod \"d85914c5-e17a-4115-afe1-b0815bcdad36\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") "
Oct 30 04:17:49.071445 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070251    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-data\") pod \"d85914c5-e17a-4115-afe1-b0815bcdad36\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") "
Oct 30 04:17:49.071445 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070306    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85914c5-e17a-4115-afe1-b0815bcdad36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85914c5-e17a-4115-afe1-b0815bcdad36-istiod-ca-cert\") pod \"d85914c5-e17a-4115-afe1-b0815bcdad36\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") "
Oct 30 04:17:49.071445 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070333    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-certs\") pod \"d85914c5-e17a-4115-afe1-b0815bcdad36\" (UID: \"d85914c5-e17a-4115-afe1-b0815bcdad36\") "
Oct 30 04:17:49.071445 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070337    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-credential-socket" pod "d85914c5-e17a-4115-afe1-b0815bcdad36" (UID: "d85914c5-e17a-4115-afe1-b0815bcdad36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:49.071445 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070522    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-socket" pod "d85914c5-e17a-4115-afe1-b0815bcdad36" (UID: "d85914c5-e17a-4115-afe1-b0815bcdad36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:49.071445 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.070650    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-data" pod "d85914c5-e17a-4115-afe1-b0815bcdad36" (UID: "d85914c5-e17a-4115-afe1-b0815bcdad36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:49.071445 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071141    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85914c5-e17a-4115-afe1-b0815bcdad36-istiod-ca-cert" pod "d85914c5-e17a-4115-afe1-b0815bcdad36" (UID: "d85914c5-e17a-4115-afe1-b0815bcdad36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:17:49.071445 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071339    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-certs" pod "d85914c5-e17a-4115-afe1-b0815bcdad36" (UID: "d85914c5-e17a-4115-afe1-b0815bcdad36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071481    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbslj\" (UniqueName: \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-kube-api-access-hbslj\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071740    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b08259-c22f-4be6-a144-4049edf81dca-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071762    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071788    4494 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071806    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlw5\" (UniqueName: \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-kube-api-access-htlw5\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071826    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071842    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f46c43-2286-48da-bb3c-acf0bd23720a-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071857    4494 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-data\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071870    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071883    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b08259-c22f-4be6-a144-4049edf81dca-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071913    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b08259-c22f-4be6-a144-4049edf81dca-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071926    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071938    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f46c43-2286-48da-bb3c-acf0bd23720a-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071951    4494 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-credential-socket\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.071984 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.071967    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b08259-c22f-4be6-a144-4049edf81dca-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.073500 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.073472    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-kube-api-access-cpn8z" pod "d85914c5-e17a-4115-afe1-b0815bcdad36" (UID: "d85914c5-e17a-4115-afe1-b0815bcdad36"). InnerVolumeSpecName "kube-api-access-cpn8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:49.073500 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.073475    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-envoy" pod "d85914c5-e17a-4115-afe1-b0815bcdad36" (UID: "d85914c5-e17a-4115-afe1-b0815bcdad36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:17:49.074304 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.074278    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-podinfo" pod "d85914c5-e17a-4115-afe1-b0815bcdad36" (UID: "d85914c5-e17a-4115-afe1-b0815bcdad36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 04:17:49.074385 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.074295    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-token" pod "d85914c5-e17a-4115-afe1-b0815bcdad36" (UID: "d85914c5-e17a-4115-afe1-b0815bcdad36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:17:49.172705 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.172601    4494 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-podinfo\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.172705 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.172653    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpn8z\" (UniqueName: \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-kube-api-access-cpn8z\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.172705 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.172665    4494 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-envoy\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.172705 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.172674    4494 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d85914c5-e17a-4115-afe1-b0815bcdad36-istiod-ca-cert\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.172705 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.172684    4494 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d85914c5-e17a-4115-afe1-b0815bcdad36-workload-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.172705 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.172692    4494 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d85914c5-e17a-4115-afe1-b0815bcdad36-istio-token\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:17:49.497542 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.497443    4494 generic.go:356] "Generic (PLEG): container finished" podID="c4b08259-c22f-4be6-a144-4049edf81dca" containerID="4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45" exitCode=0
Oct 30 04:17:49.497542 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.497497    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"
Oct 30 04:17:49.497542 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.497527    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" event={"ID":"c4b08259-c22f-4be6-a144-4049edf81dca","Type":"ContainerDied","Data":"4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45"}
Oct 30 04:17:49.497936 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.497586    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6" event={"ID":"c4b08259-c22f-4be6-a144-4049edf81dca","Type":"ContainerDied","Data":"4eee5d6421844a76c24f57e06993a7fc5df71173cd8e471bf0b55bb8425699ac"}
Oct 30 04:17:49.497936 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.497619    4494 scope.go:122] "RemoveContainer" containerID="4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45"
Oct 30 04:17:49.499119 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.499085    4494 generic.go:356] "Generic (PLEG): container finished" podID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerID="9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39" exitCode=0
Oct 30 04:17:49.499215 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.499163    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"
Oct 30 04:17:49.499215 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.499177    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" event={"ID":"33f46c43-2286-48da-bb3c-acf0bd23720a","Type":"ContainerDied","Data":"9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39"}
Oct 30 04:17:49.499326 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.499240    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2" event={"ID":"33f46c43-2286-48da-bb3c-acf0bd23720a","Type":"ContainerDied","Data":"096efc6b9b3865ff892a066026b431274e6e936a99629af67daef5d94cf5e9ec"}
Oct 30 04:17:49.500698 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.500670    4494 generic.go:356] "Generic (PLEG): container finished" podID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerID="a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7" exitCode=0
Oct 30 04:17:49.500832 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.500719    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"
Oct 30 04:17:49.500832 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.500748    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" event={"ID":"d85914c5-e17a-4115-afe1-b0815bcdad36","Type":"ContainerDied","Data":"a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7"}
Oct 30 04:17:49.500832 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.500811    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc" event={"ID":"d85914c5-e17a-4115-afe1-b0815bcdad36","Type":"ContainerDied","Data":"be584af147691ba5530af6cc8abefe92e2c67327df7c767429bf5d639915d4ec"}
Oct 30 04:17:49.507379 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.507353    4494 scope.go:122] "RemoveContainer" containerID="4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45"
Oct 30 04:17:49.507695 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:17:49.507670    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45\": container with ID starting with 4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45 not found: ID does not exist" containerID="4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45"
Oct 30 04:17:49.507794 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.507704    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45"} err="failed to get container status \"4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45\": rpc error: code = NotFound desc = could not find container \"4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45\": container with ID starting with 4cf33dee7a17d521d4573d2f037e4ad488d3fad80e4602bed6d19f12c1922e45 not found: ID does not exist"
Oct 30 04:17:49.507794 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.507724    4494 scope.go:122] "RemoveContainer" containerID="9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39"
Oct 30 04:17:49.517539 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.517518    4494 scope.go:122] "RemoveContainer" containerID="9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39"
Oct 30 04:17:49.517874 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:17:49.517852    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39\": container with ID starting with 9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39 not found: ID does not exist" containerID="9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39"
Oct 30 04:17:49.518075 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.517887    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39"} err="failed to get container status \"9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39\": rpc error: code = NotFound desc = could not find container \"9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39\": container with ID starting with 9389e1296f0eb4439a09700cef4101a5f4f080697d5b65d2863594817d37cb39 not found: ID does not exist"
Oct 30 04:17:49.518075 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.517928    4494 scope.go:122] "RemoveContainer" containerID="a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7"
Oct 30 04:17:49.526608 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.526581    4494 scope.go:122] "RemoveContainer" containerID="a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7"
Oct 30 04:17:49.526938 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:17:49.526913    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7\": container with ID starting with a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7 not found: ID does not exist" containerID="a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7"
Oct 30 04:17:49.527008 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.526952    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7"} err="failed to get container status \"a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7\": rpc error: code = NotFound desc = could not find container \"a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7\": container with ID starting with a3913a4ea6987b6dc749255fcd7c3652a741ad27be3db5930ce01ea3eba441f7 not found: ID does not exist"
Oct 30 04:17:49.579533 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.579490    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"]
Oct 30 04:17:49.592135 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.592103    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-jzwm6"]
Oct 30 04:17:49.626475 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.626436    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"]
Oct 30 04:17:49.630689 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.630655    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-25jp2"]
Oct 30 04:17:49.665339 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.665299    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"]
Oct 30 04:17:49.669059 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.669024    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-smtlc"]
Oct 30 04:17:49.927468 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.927433    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" path="/var/lib/kubelet/pods/33f46c43-2286-48da-bb3c-acf0bd23720a/volumes"
Oct 30 04:17:49.927969 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.927954    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" path="/var/lib/kubelet/pods/c4b08259-c22f-4be6-a144-4049edf81dca/volumes"
Oct 30 04:17:49.928396 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:49.928382    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" path="/var/lib/kubelet/pods/d85914c5-e17a-4115-afe1-b0815bcdad36/volumes"
Oct 30 04:17:54.004923 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.004869    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"]
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005251    4494 cpu_manager.go:414] "containerMap: removing container" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" containerName="istio-proxy"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005263    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" containerName="istio-proxy"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005276    4494 cpu_manager.go:414] "containerMap: removing container" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="storage-initializer"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005281    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="storage-initializer"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005287    4494 cpu_manager.go:414] "containerMap: removing container" podUID="168bf7fe-7e7c-4cfe-b73e-e0d90960353a" containerName="storage-initializer"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005293    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168bf7fe-7e7c-4cfe-b73e-e0d90960353a" containerName="storage-initializer"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005300    4494 cpu_manager.go:414] "containerMap: removing container" podUID="168bf7fe-7e7c-4cfe-b73e-e0d90960353a" containerName="main"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005304    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168bf7fe-7e7c-4cfe-b73e-e0d90960353a" containerName="main"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005313    4494 cpu_manager.go:414] "containerMap: removing container" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerName="istio-proxy"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005317    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerName="istio-proxy"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005331    4494 cpu_manager.go:414] "containerMap: removing container" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005336    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005343    4494 cpu_manager.go:414] "containerMap: removing container" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="llm-d-routing-sidecar"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005347    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="llm-d-routing-sidecar"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005356    4494 cpu_manager.go:414] "containerMap: removing container" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerName="istio-proxy"
Oct 30 04:17:54.005392 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005360    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerName="istio-proxy"
Oct 30 04:17:54.005875 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005412    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b08259-c22f-4be6-a144-4049edf81dca" containerName="istio-proxy"
Oct 30 04:17:54.005875 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005421    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="168bf7fe-7e7c-4cfe-b73e-e0d90960353a" containerName="main"
Oct 30 04:17:54.005875 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005427    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85914c5-e17a-4115-afe1-b0815bcdad36" containerName="istio-proxy"
Oct 30 04:17:54.005875 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005434    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="main"
Oct 30 04:17:54.005875 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005439    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="50816443-65a5-4317-9da2-7592ab3ef5f2" containerName="llm-d-routing-sidecar"
Oct 30 04:17:54.005875 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.005444    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f46c43-2286-48da-bb3c-acf0bd23720a" containerName="istio-proxy"
Oct 30 04:17:54.010113 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.010088    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.016644 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.016614    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.016811 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.016652    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.016811 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.016672    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.016960 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.016803    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g66p\" (UniqueName: \"kubernetes.io/projected/23cf63a6-e63a-4b06-9511-4d243425e553-kube-api-access-8g66p\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.016960 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.016840    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.016960 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.016863    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cf63a6-e63a-4b06-9511-4d243425e553-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.017132 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.016976    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cf63a6-e63a-4b06-9511-4d243425e553-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.017132 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.017010    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.017132 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.017076    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cf63a6-e63a-4b06-9511-4d243425e553-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.025073 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.025049    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 04:17:54.037220 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.037187    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 04:17:54.037220 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.037188    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 04:17:54.037442 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.037205    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-pzw79\""
Oct 30 04:17:54.039094 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.039071    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"]
Oct 30 04:17:54.083180 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:17:54.083140    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:17:54.117654 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.117621    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cf63a6-e63a-4b06-9511-4d243425e553-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.117654 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.117665    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.117938 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.117688    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cf63a6-e63a-4b06-9511-4d243425e553-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.117938 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.117714    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.117938 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.117839    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.117938 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.117866    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.118175 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.117953    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g66p\" (UniqueName: \"kubernetes.io/projected/23cf63a6-e63a-4b06-9511-4d243425e553-kube-api-access-8g66p\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.118175 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.117995    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.118175 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.118029    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cf63a6-e63a-4b06-9511-4d243425e553-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.118350 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.118224    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.118350 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.118259    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.118500 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.118480    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.118606 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.118575    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.118920 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.118872    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cf63a6-e63a-4b06-9511-4d243425e553-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.120402 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.120372    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cf63a6-e63a-4b06-9511-4d243425e553-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.120504 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.120490    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cf63a6-e63a-4b06-9511-4d243425e553-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.141857 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.141818    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cf63a6-e63a-4b06-9511-4d243425e553-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.142036 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.141868    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g66p\" (UniqueName: \"kubernetes.io/projected/23cf63a6-e63a-4b06-9511-4d243425e553-kube-api-access-8g66p\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql\" (UID: \"23cf63a6-e63a-4b06-9511-4d243425e553\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.324110 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.324073    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:54.634731 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.634700    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"]
Oct 30 04:17:54.635161 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:17:54.635135    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cf63a6_e63a_4b06_9511_4d243425e553.slice/crio-c93b06c997abac4e00f2195bb1c401770513639fde3d4ee60cc9098a62f1ca7a WatchSource:0}: Error finding container c93b06c997abac4e00f2195bb1c401770513639fde3d4ee60cc9098a62f1ca7a: Status 404 returned error can't find the container with id c93b06c997abac4e00f2195bb1c401770513639fde3d4ee60cc9098a62f1ca7a
Oct 30 04:17:54.637164 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.637131    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:17:54.637271 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.637207    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:17:54.637271 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:54.637252    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:17:55.534125 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:55.534091    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql" event={"ID":"23cf63a6-e63a-4b06-9511-4d243425e553","Type":"ContainerStarted","Data":"b2933f9c9448f7a7e8c959779974bf2aa3d9eb5b4ef014d891385081abb18f10"}
Oct 30 04:17:55.534568 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:55.534147    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql" event={"ID":"23cf63a6-e63a-4b06-9511-4d243425e553","Type":"ContainerStarted","Data":"c93b06c997abac4e00f2195bb1c401770513639fde3d4ee60cc9098a62f1ca7a"}
Oct 30 04:17:55.605876 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:55.605804    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql" podStartSLOduration=2.605781216 podStartE2EDuration="2.605781216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:55.592220998 +0000 UTC m=+4939.798401217" watchObservedRunningTime="2026-10-30 04:17:55.605781216 +0000 UTC m=+4939.811961435"
Oct 30 04:17:56.325014 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:56.324966    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:17:56.326555 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:56.326527    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 04:17:56.326672 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:56.326587    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql" podUID="23cf63a6-e63a-4b06-9511-4d243425e553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 04:17:57.324521 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:57.324474    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 04:17:57.325021 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:57.324544    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql" podUID="23cf63a6-e63a-4b06-9511-4d243425e553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 04:17:58.325526 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:58.325479    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 04:17:58.326005 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:58.325542    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql" podUID="23cf63a6-e63a-4b06-9511-4d243425e553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 04:17:59.324300 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:59.324261    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 04:17:59.324480 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:17:59.324316    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql" podUID="23cf63a6-e63a-4b06-9511-4d243425e553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 04:18:00.324873 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:00.324829    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 04:18:00.325335 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:00.324926    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql" podUID="23cf63a6-e63a-4b06-9511-4d243425e553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 04:18:01.329159 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:01.329126    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:18:01.329581 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:01.329404    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:18:01.330244 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:01.330227    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql"
Oct 30 04:18:09.054459 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.054420    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"]
Oct 30 04:18:09.058232 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.058207    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.073217 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.073185    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"]
Oct 30 04:18:09.076688 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.076664    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.082970 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.082938    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"]
Oct 30 04:18:09.104457 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.104417    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"]
Oct 30 04:18:09.140003 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.139963    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.140003 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140009    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.140231 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140030    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.140231 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140113    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.140301 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140222    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.140301 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140268    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.140301 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140289    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hs9\" (UniqueName: \"kubernetes.io/projected/efe3c14d-9bd0-4153-a117-db52d9dae3ee-kube-api-access-27hs9\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.140399 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140310    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.140399 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140392    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.140464 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140417    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.140464 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140439    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.140532 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140463    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.140532 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140512    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.140598 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140541    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6653351-672f-4b10-bcca-68f6cfce20f6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.140598 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140580    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4n7\" (UniqueName: \"kubernetes.io/projected/d6653351-672f-4b10-bcca-68f6cfce20f6-kube-api-access-xs4n7\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.140663 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140604    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.140663 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140622    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.140663 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.140637    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.241861 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.241813    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.241861 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.241864    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.242168 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.241881    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6653351-672f-4b10-bcca-68f6cfce20f6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.242168 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.242095    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4n7\" (UniqueName: \"kubernetes.io/projected/d6653351-672f-4b10-bcca-68f6cfce20f6-kube-api-access-xs4n7\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.242168 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.242147    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.242339 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.242173    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.242339 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.242217    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.242339 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.242291    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.242475 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.242337    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.243183 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243152    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.243513 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243298    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.243607 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243555    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.243607 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243581    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.243713 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243605    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.243713 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243663    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.243713 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243695    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hs9\" (UniqueName: \"kubernetes.io/projected/efe3c14d-9bd0-4153-a117-db52d9dae3ee-kube-api-access-27hs9\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.243873 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243734    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.243873 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243777    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.243873 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243820    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.244052 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243873    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.244052 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243891    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.244052 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243918    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6653351-672f-4b10-bcca-68f6cfce20f6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.244052 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.243415    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.244241 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.244153    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.244420 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.244342    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.246849 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.244679    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.246849 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.244692    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.247080 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.247061    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.249931 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.247643    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.249931 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.247665    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.252401 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.252372    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.252518 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.252400    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.265852 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.265749    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe3c14d-9bd0-4153-a117-db52d9dae3ee-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.266306 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.266274    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4n7\" (UniqueName: \"kubernetes.io/projected/d6653351-672f-4b10-bcca-68f6cfce20f6-kube-api-access-xs4n7\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.267513 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.267489    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hs9\" (UniqueName: \"kubernetes.io/projected/efe3c14d-9bd0-4153-a117-db52d9dae3ee-kube-api-access-27hs9\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2\" (UID: \"efe3c14d-9bd0-4153-a117-db52d9dae3ee\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.267628 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.267597    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6653351-672f-4b10-bcca-68f6cfce20f6-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7\" (UID: \"d6653351-672f-4b10-bcca-68f6cfce20f6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.372640 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.372539    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:09.392102 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.392070    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:09.683879 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.683767    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"]
Oct 30 04:18:09.691353 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:18:09.691305    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe3c14d_9bd0_4153_a117_db52d9dae3ee.slice/crio-3171a1a7c0462d07d1f917fa65ffdd28282b6c0be6f1510c963edf1b9bdd9323 WatchSource:0}: Error finding container 3171a1a7c0462d07d1f917fa65ffdd28282b6c0be6f1510c963edf1b9bdd9323: Status 404 returned error can't find the container with id 3171a1a7c0462d07d1f917fa65ffdd28282b6c0be6f1510c963edf1b9bdd9323
Oct 30 04:18:09.693922 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.693853    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:18:09.694019 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.693997    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:18:09.694063 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.694048    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:18:09.705610 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.705150    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"]
Oct 30 04:18:09.708602 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:18:09.708565    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6653351_672f_4b10_bcca_68f6cfce20f6.slice/crio-9208721f72991ce9600f45d2adcc823eef5d7a5fb7a33ef22db7848a5851af48 WatchSource:0}: Error finding container 9208721f72991ce9600f45d2adcc823eef5d7a5fb7a33ef22db7848a5851af48: Status 404 returned error can't find the container with id 9208721f72991ce9600f45d2adcc823eef5d7a5fb7a33ef22db7848a5851af48
Oct 30 04:18:09.711008 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.710971    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:18:09.711123 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.711045    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:18:09.711123 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:09.711077    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:18:10.627923 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:10.627558    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7" event={"ID":"d6653351-672f-4b10-bcca-68f6cfce20f6","Type":"ContainerStarted","Data":"e2256c464d20816add61fb1d17d731ed72ac3835116fe5631bba5093dfddb44e"}
Oct 30 04:18:10.627923 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:10.627630    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7" event={"ID":"d6653351-672f-4b10-bcca-68f6cfce20f6","Type":"ContainerStarted","Data":"9208721f72991ce9600f45d2adcc823eef5d7a5fb7a33ef22db7848a5851af48"}
Oct 30 04:18:10.630809 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:10.630770    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2" event={"ID":"efe3c14d-9bd0-4153-a117-db52d9dae3ee","Type":"ContainerStarted","Data":"f04ff6eebed2fac79010c400521904dd5a914492c15622b5d798dd4424c38774"}
Oct 30 04:18:10.631111 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:10.631079    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2" event={"ID":"efe3c14d-9bd0-4153-a117-db52d9dae3ee","Type":"ContainerStarted","Data":"3171a1a7c0462d07d1f917fa65ffdd28282b6c0be6f1510c963edf1b9bdd9323"}
Oct 30 04:18:10.726002 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:10.725933    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7" podStartSLOduration=1.725892679 podStartE2EDuration="1.725892679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:10.701408195 +0000 UTC m=+4954.907588414" watchObservedRunningTime="2026-10-30 04:18:10.725892679 +0000 UTC m=+4954.932072902"
Oct 30 04:18:10.758077 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:10.758017    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2" podStartSLOduration=1.758004229 podStartE2EDuration="1.758004229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:10.75712162 +0000 UTC m=+4954.963301840" watchObservedRunningTime="2026-10-30 04:18:10.758004229 +0000 UTC m=+4954.964184446"
Oct 30 04:18:11.373522 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:11.373445    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:11.374942 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:11.374889    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 04:18:11.375079 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:11.374967    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2" podUID="efe3c14d-9bd0-4153-a117-db52d9dae3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 04:18:11.399561 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:11.399482    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:11.411924 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:11.402316    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 04:18:11.411924 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:11.402393    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7" podUID="d6653351-672f-4b10-bcca-68f6cfce20f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 04:18:12.128476 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.128416    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"]
Oct 30 04:18:12.133658 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.133619    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.166180 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.166136    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Oct 30 04:18:12.166180 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.166167    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Oct 30 04:18:12.166180 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.166180    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Oct 30 04:18:12.166423 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.166155    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-sgm5x\""
Oct 30 04:18:12.166423 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.166299    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"]
Oct 30 04:18:12.175166 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.175131    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-home\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.175295 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.175174    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-dshm\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.175295 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.175276    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dhxx\" (UniqueName: \"kubernetes.io/projected/8e6d345b-2d98-45a5-adeb-085b42266c5c-kube-api-access-9dhxx\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.175368 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.175321    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6d345b-2d98-45a5-adeb-085b42266c5c-tls-certs\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.175423 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.175405    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-model-cache\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.175465 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.175444    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-tmp-dir\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.276957 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.276876    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dhxx\" (UniqueName: \"kubernetes.io/projected/8e6d345b-2d98-45a5-adeb-085b42266c5c-kube-api-access-9dhxx\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.277156 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.276976    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6d345b-2d98-45a5-adeb-085b42266c5c-tls-certs\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.277156 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.277018    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-model-cache\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.277156 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.277038    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-tmp-dir\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.277156 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.277081    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-home\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.277156 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.277103    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-dshm\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.277796 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.277558    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-model-cache\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.277796 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.277745    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-home\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.277985 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.277812    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-tmp-dir\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.292278 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.292236    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-dshm\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.305643 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.305606    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6d345b-2d98-45a5-adeb-085b42266c5c-tls-certs\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.309931 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.308635    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dhxx\" (UniqueName: \"kubernetes.io/projected/8e6d345b-2d98-45a5-adeb-085b42266c5c-kube-api-access-9dhxx\") pod \"rolling-upgrade-test-kserve-78f4d666cb-nwk4s\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.373962 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.373785    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 04:18:12.373962 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.373838    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2" podUID="efe3c14d-9bd0-4153-a117-db52d9dae3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 04:18:12.400193 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.399994    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 04:18:12.400193 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.400067    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7" podUID="d6653351-672f-4b10-bcca-68f6cfce20f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 04:18:12.446916 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.446869    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:12.803974 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:12.795048    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"]
Oct 30 04:18:13.373775 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:13.373740    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 04:18:13.374231 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:13.373793    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2" podUID="efe3c14d-9bd0-4153-a117-db52d9dae3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 04:18:13.399422 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:13.399385    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 04:18:13.399688 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:13.399658    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7" podUID="d6653351-672f-4b10-bcca-68f6cfce20f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 04:18:13.646644 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:13.646549    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s" event={"ID":"8e6d345b-2d98-45a5-adeb-085b42266c5c","Type":"ContainerStarted","Data":"00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb"}
Oct 30 04:18:13.646644 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:13.646636    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s" event={"ID":"8e6d345b-2d98-45a5-adeb-085b42266c5c","Type":"ContainerStarted","Data":"31d27d0ea45592f2cc79c3ba128927adb71c9c99a2abd10c2a4e22aab5ee38a2"}
Oct 30 04:18:13.697173 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:13.697104    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s" podStartSLOduration=1.697083901 podStartE2EDuration="1.697083901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:13.696384424 +0000 UTC m=+4957.902564644" watchObservedRunningTime="2026-10-30 04:18:13.697083901 +0000 UTC m=+4957.903264120"
Oct 30 04:18:14.373510 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:14.373466    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 04:18:14.373696 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:14.373528    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2" podUID="efe3c14d-9bd0-4153-a117-db52d9dae3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 04:18:14.399567 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:14.399521    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 04:18:14.400304 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:14.400259    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7" podUID="d6653351-672f-4b10-bcca-68f6cfce20f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 04:18:14.447328 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:14.447285    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:15.373443 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:15.373407    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 04:18:15.373605 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:15.373464    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2" podUID="efe3c14d-9bd0-4153-a117-db52d9dae3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 04:18:15.397173 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:15.397124    4494 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 04:18:15.397503 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:15.397455    4494 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7" podUID="d6653351-672f-4b10-bcca-68f6cfce20f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 04:18:16.338371 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:16.338335    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"]
Oct 30 04:18:16.377997 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:16.377756    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:16.378385 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:16.378360    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:16.379187 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:16.379166    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2"
Oct 30 04:18:16.396571 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:16.396542    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:16.396856 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:16.396829    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:16.397659 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:16.397639    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7"
Oct 30 04:18:16.672418 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:18:16.672371    4494 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb" cmd=["/bin/sleep","15"]
Oct 30 04:18:16.672575 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:16.672433    4494 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 04:18:16.672575 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:18:16.672450    4494 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s" podUID="8e6d345b-2d98-45a5-adeb-085b42266c5c" containerName="main" containerID="cri-o://00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb"
Oct 30 04:18:16.672575 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:16.672493    4494 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s" podUID="8e6d345b-2d98-45a5-adeb-085b42266c5c" containerName="main" containerID="cri-o://00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb" gracePeriod=60
Oct 30 04:18:16.921381 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:16.921352    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:17.070435 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.070391    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-model-cache\") pod \"8e6d345b-2d98-45a5-adeb-085b42266c5c\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") "
Oct 30 04:18:17.070651 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.070447    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-dshm\") pod \"8e6d345b-2d98-45a5-adeb-085b42266c5c\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") "
Oct 30 04:18:17.070651 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.070489    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-home\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-home\") pod \"8e6d345b-2d98-45a5-adeb-085b42266c5c\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") "
Oct 30 04:18:17.070651 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.070515    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-tmp-dir\") pod \"8e6d345b-2d98-45a5-adeb-085b42266c5c\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") "
Oct 30 04:18:17.070651 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.070614    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6d345b-2d98-45a5-adeb-085b42266c5c-kube-api-access-9dhxx\" (UniqueName: \"kubernetes.io/projected/8e6d345b-2d98-45a5-adeb-085b42266c5c-kube-api-access-9dhxx\") pod \"8e6d345b-2d98-45a5-adeb-085b42266c5c\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") "
Oct 30 04:18:17.070651 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.070635    4494 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6d345b-2d98-45a5-adeb-085b42266c5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6d345b-2d98-45a5-adeb-085b42266c5c-tls-certs\") pod \"8e6d345b-2d98-45a5-adeb-085b42266c5c\" (UID: \"8e6d345b-2d98-45a5-adeb-085b42266c5c\") "
Oct 30 04:18:17.070947 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.070720    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-model-cache" pod "8e6d345b-2d98-45a5-adeb-085b42266c5c" (UID: "8e6d345b-2d98-45a5-adeb-085b42266c5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:18:17.070947 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.070777    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-home" pod "8e6d345b-2d98-45a5-adeb-085b42266c5c" (UID: "8e6d345b-2d98-45a5-adeb-085b42266c5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:18:17.070947 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.070851    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-tmp-dir" pod "8e6d345b-2d98-45a5-adeb-085b42266c5c" (UID: "8e6d345b-2d98-45a5-adeb-085b42266c5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:18:17.071059 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.070985    4494 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-home\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.071059 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.071005    4494 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-tmp-dir\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.071059 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.071027    4494 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-model-cache\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.073192 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.073165    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6d345b-2d98-45a5-adeb-085b42266c5c-tls-certs" pod "8e6d345b-2d98-45a5-adeb-085b42266c5c" (UID: "8e6d345b-2d98-45a5-adeb-085b42266c5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:18:17.073306 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.073190    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-dshm" pod "8e6d345b-2d98-45a5-adeb-085b42266c5c" (UID: "8e6d345b-2d98-45a5-adeb-085b42266c5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:18:17.073306 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.073208    4494 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6d345b-2d98-45a5-adeb-085b42266c5c-kube-api-access-9dhxx" pod "8e6d345b-2d98-45a5-adeb-085b42266c5c" (UID: "8e6d345b-2d98-45a5-adeb-085b42266c5c"). InnerVolumeSpecName "kube-api-access-9dhxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:18:17.172119 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.172018    4494 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dhxx\" (UniqueName: \"kubernetes.io/projected/8e6d345b-2d98-45a5-adeb-085b42266c5c-kube-api-access-9dhxx\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.172119 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.172055    4494 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6d345b-2d98-45a5-adeb-085b42266c5c-tls-certs\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.172119 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.172064    4494 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6d345b-2d98-45a5-adeb-085b42266c5c-dshm\") on node \"ip-10-0-130-130.ec2.internal\" DevicePath \"\""
Oct 30 04:18:17.668975 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.668936    4494 generic.go:356] "Generic (PLEG): container finished" podID="8e6d345b-2d98-45a5-adeb-085b42266c5c" containerID="00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb" exitCode=0
Oct 30 04:18:17.669505 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.669043    4494 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"
Oct 30 04:18:17.669505 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.669062    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s" event={"ID":"8e6d345b-2d98-45a5-adeb-085b42266c5c","Type":"ContainerDied","Data":"00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb"}
Oct 30 04:18:17.669505 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.669110    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s" event={"ID":"8e6d345b-2d98-45a5-adeb-085b42266c5c","Type":"ContainerDied","Data":"31d27d0ea45592f2cc79c3ba128927adb71c9c99a2abd10c2a4e22aab5ee38a2"}
Oct 30 04:18:17.669505 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.669135    4494 scope.go:122] "RemoveContainer" containerID="00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb"
Oct 30 04:18:17.679127 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.678990    4494 scope.go:122] "RemoveContainer" containerID="00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb"
Oct 30 04:18:17.679305 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:18:17.679285    4494 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb\": container with ID starting with 00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb not found: ID does not exist" containerID="00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb"
Oct 30 04:18:17.679341 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.679313    4494 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb"} err="failed to get container status \"00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb\": rpc error: code = NotFound desc = could not find container \"00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb\": container with ID starting with 00f65e67a469d5b1d8c6f16ede2aaa8b7f824d721ad5ee79ee84b7809c39aabb not found: ID does not exist"
Oct 30 04:18:17.720656 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.720619    4494 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"]
Oct 30 04:18:17.732642 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.732606    4494 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-78f4d666cb-nwk4s"]
Oct 30 04:18:17.927708 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:17.927616    4494 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6d345b-2d98-45a5-adeb-085b42266c5c" path="/var/lib/kubelet/pods/8e6d345b-2d98-45a5-adeb-085b42266c5c/volumes"
Oct 30 04:18:24.083165 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:18:24.083117    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:18:24.150523 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.150482    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"]
Oct 30 04:18:24.150954 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.150938    4494 cpu_manager.go:414] "containerMap: removing container" podUID="8e6d345b-2d98-45a5-adeb-085b42266c5c" containerName="main"
Oct 30 04:18:24.151011 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.150955    4494 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6d345b-2d98-45a5-adeb-085b42266c5c" containerName="main"
Oct 30 04:18:24.151047 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.151026    4494 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6d345b-2d98-45a5-adeb-085b42266c5c" containerName="main"
Oct 30 04:18:24.154367 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.154342    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.183448 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.183412    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"]
Oct 30 04:18:24.231106 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.231058    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.231336 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.231122    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d37115d-21f2-4744-9bbc-f86e83367b98-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.231336 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.231197    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.231336 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.231226    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.231336 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.231264    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.231336 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.231310    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5lms\" (UniqueName: \"kubernetes.io/projected/0d37115d-21f2-4744-9bbc-f86e83367b98-kube-api-access-m5lms\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.231564 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.231358    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.231564 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.231389    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.231564 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.231457    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.332425 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.332382    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d37115d-21f2-4744-9bbc-f86e83367b98-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.332425 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.332422    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.332683 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.332442    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.332683 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.332466    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.332683 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.332485    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5lms\" (UniqueName: \"kubernetes.io/projected/0d37115d-21f2-4744-9bbc-f86e83367b98-kube-api-access-m5lms\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.332683 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.332510    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.332683 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.332536    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.332683 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.332579    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.332683 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.332638    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.333034 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.332888    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-workload-certs\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.333034 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.333009    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-credential-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.333109 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.333047    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-workload-socket\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.333303 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.333259    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d37115d-21f2-4744-9bbc-f86e83367b98-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.333345 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.333296    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-data\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.335055 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.335037    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.335365 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.335346    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.356049 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.356016    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d37115d-21f2-4744-9bbc-f86e83367b98-istio-token\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.356261 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.356235    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5lms\" (UniqueName: \"kubernetes.io/projected/0d37115d-21f2-4744-9bbc-f86e83367b98-kube-api-access-m5lms\") pod \"openshift-ai-inference-openshift-default-5b4f8cc59b-png5f\" (UID: \"0d37115d-21f2-4744-9bbc-f86e83367b98\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.469086 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.469051    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:24.781536 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.781501    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"]
Oct 30 04:18:24.781944 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:18:24.781880    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d37115d_21f2_4744_9bbc_f86e83367b98.slice/crio-51b093d81dfc83db86b417170693bf5bc0e25b2982a6f1e7818128f0144bd95d WatchSource:0}: Error finding container 51b093d81dfc83db86b417170693bf5bc0e25b2982a6f1e7818128f0144bd95d: Status 404 returned error can't find the container with id 51b093d81dfc83db86b417170693bf5bc0e25b2982a6f1e7818128f0144bd95d
Oct 30 04:18:24.784210 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.784177    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:18:24.784313 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.784262    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:18:24.784313 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:24.784305    4494 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 30 04:18:25.712966 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:25.712921    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f" event={"ID":"0d37115d-21f2-4744-9bbc-f86e83367b98","Type":"ContainerStarted","Data":"b3739c70f6fa78e3280d38361d87205526aee55101fae3e9ff4a725189084d9d"}
Oct 30 04:18:25.713482 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:25.712982    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f" event={"ID":"0d37115d-21f2-4744-9bbc-f86e83367b98","Type":"ContainerStarted","Data":"51b093d81dfc83db86b417170693bf5bc0e25b2982a6f1e7818128f0144bd95d"}
Oct 30 04:18:25.770561 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:25.770505    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f" podStartSLOduration=1.7704898070000001 podStartE2EDuration="1.770489807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:25.770057975 +0000 UTC m=+4969.976238193" watchObservedRunningTime="2026-10-30 04:18:25.770489807 +0000 UTC m=+4969.976670024"
Oct 30 04:18:26.470019 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:26.469965    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:26.475341 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:26.475315    4494 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:26.717170 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:26.717134    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:26.718280 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:18:26.718256    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5b4f8cc59b-png5f"
Oct 30 04:18:54.082052 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:18:54.081967    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:19:24.082262 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:19:24.082219    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:19:44.649397 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:44.649355    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7_d6653351-672f-4b10-bcca-68f6cfce20f6/istio-proxy/0.log"
Oct 30 04:19:44.765093 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:44.765067    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql_23cf63a6-e63a-4b06-9511-4d243425e553/istio-proxy/0.log"
Oct 30 04:19:44.960231 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:44.960144    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2_efe3c14d-9bd0-4153-a117-db52d9dae3ee/istio-proxy/0.log"
Oct 30 04:19:45.026522 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:45.026494    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-png5f_0d37115d-21f2-4744-9bbc-f86e83367b98/istio-proxy/0.log"
Oct 30 04:19:47.086569 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:47.086534    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7_d6653351-672f-4b10-bcca-68f6cfce20f6/istio-proxy/0.log"
Oct 30 04:19:47.149849 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:47.149823    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql_23cf63a6-e63a-4b06-9511-4d243425e553/istio-proxy/0.log"
Oct 30 04:19:47.343279 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:47.343204    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2_efe3c14d-9bd0-4153-a117-db52d9dae3ee/istio-proxy/0.log"
Oct 30 04:19:47.406176 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:47.406146    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-png5f_0d37115d-21f2-4744-9bbc-f86e83367b98/istio-proxy/0.log"
Oct 30 04:19:49.001201 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:49.001168    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-557c8f8584-6lfw5_06f8e1b8-8ddd-4298-9c61-f326e03387a7/authorino/0.log"
Oct 30 04:19:52.746746 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.746713    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lgh6m/must-gather-8bqdp"]
Oct 30 04:19:52.750206 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.750190    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgh6m/must-gather-8bqdp"
Oct 30 04:19:52.779675 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.779650    4494 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lgh6m\"/\"default-dockercfg-bgns5\""
Oct 30 04:19:52.779675 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.779647    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lgh6m\"/\"kube-root-ca.crt\""
Oct 30 04:19:52.779869 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.779658    4494 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lgh6m\"/\"openshift-service-ca.crt\""
Oct 30 04:19:52.781218 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.781197    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgh6m/must-gather-8bqdp"]
Oct 30 04:19:52.853238 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.853204    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cfcb-d622-463e-ba29-af9d90393035-must-gather-output\") pod \"must-gather-8bqdp\" (UID: \"5cf5cfcb-d622-463e-ba29-af9d90393035\") " pod="openshift-must-gather-lgh6m/must-gather-8bqdp"
Oct 30 04:19:52.853409 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.853260    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f6ws\" (UniqueName: \"kubernetes.io/projected/5cf5cfcb-d622-463e-ba29-af9d90393035-kube-api-access-8f6ws\") pod \"must-gather-8bqdp\" (UID: \"5cf5cfcb-d622-463e-ba29-af9d90393035\") " pod="openshift-must-gather-lgh6m/must-gather-8bqdp"
Oct 30 04:19:52.953840 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.953806    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cfcb-d622-463e-ba29-af9d90393035-must-gather-output\") pod \"must-gather-8bqdp\" (UID: \"5cf5cfcb-d622-463e-ba29-af9d90393035\") " pod="openshift-must-gather-lgh6m/must-gather-8bqdp"
Oct 30 04:19:52.954041 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.953853    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f6ws\" (UniqueName: \"kubernetes.io/projected/5cf5cfcb-d622-463e-ba29-af9d90393035-kube-api-access-8f6ws\") pod \"must-gather-8bqdp\" (UID: \"5cf5cfcb-d622-463e-ba29-af9d90393035\") " pod="openshift-must-gather-lgh6m/must-gather-8bqdp"
Oct 30 04:19:52.954178 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.954159    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5cf5cfcb-d622-463e-ba29-af9d90393035-must-gather-output\") pod \"must-gather-8bqdp\" (UID: \"5cf5cfcb-d622-463e-ba29-af9d90393035\") " pod="openshift-must-gather-lgh6m/must-gather-8bqdp"
Oct 30 04:19:52.991132 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:52.991102    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f6ws\" (UniqueName: \"kubernetes.io/projected/5cf5cfcb-d622-463e-ba29-af9d90393035-kube-api-access-8f6ws\") pod \"must-gather-8bqdp\" (UID: \"5cf5cfcb-d622-463e-ba29-af9d90393035\") " pod="openshift-must-gather-lgh6m/must-gather-8bqdp"
Oct 30 04:19:53.063509 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:53.063479    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgh6m/must-gather-8bqdp"
Oct 30 04:19:53.354701 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:19:53.354668    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cf5cfcb_d622_463e_ba29_af9d90393035.slice/crio-a8c4bb0f693fa84e6449b64a9e386d531c82a9bf2b2a886435abd5a9c47f425f WatchSource:0}: Error finding container a8c4bb0f693fa84e6449b64a9e386d531c82a9bf2b2a886435abd5a9c47f425f: Status 404 returned error can't find the container with id a8c4bb0f693fa84e6449b64a9e386d531c82a9bf2b2a886435abd5a9c47f425f
Oct 30 04:19:53.355220 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:53.355195    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgh6m/must-gather-8bqdp"]
Oct 30 04:19:53.356510 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:53.356487    4494 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 04:19:54.087597 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:19:54.087554    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:19:54.112684 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:54.112647    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgh6m/must-gather-8bqdp" event={"ID":"5cf5cfcb-d622-463e-ba29-af9d90393035","Type":"ContainerStarted","Data":"a8c4bb0f693fa84e6449b64a9e386d531c82a9bf2b2a886435abd5a9c47f425f"}
Oct 30 04:19:55.120455 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:55.119819    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgh6m/must-gather-8bqdp" event={"ID":"5cf5cfcb-d622-463e-ba29-af9d90393035","Type":"ContainerStarted","Data":"2d510120db06f025e61c2597c187fca37b85a22b09a175f05a3a58fb5e8bc306"}
Oct 30 04:19:55.120455 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:55.119879    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgh6m/must-gather-8bqdp" event={"ID":"5cf5cfcb-d622-463e-ba29-af9d90393035","Type":"ContainerStarted","Data":"1d3b3da557ae036828e1a98ae17513e0aa4a9919251598f861ff87438b11c5ee"}
Oct 30 04:19:55.162451 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:55.162382    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lgh6m/must-gather-8bqdp" podStartSLOduration=2.340185764 podStartE2EDuration="3.162361862s" totalImagesPullingTime="822.176098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:55.161838132 +0000 UTC m=+5059.368018350" watchObservedRunningTime="2026-10-30 04:19:55.162361862 +0000 UTC m=+5059.368542081"
Oct 30 04:19:57.244584 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:57.244554    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ddlpl_0d303f31-40f2-46fc-9bb1-73ba2851a5ec/global-pull-secret-syncer/0.log"
Oct 30 04:19:58.314558 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:58.314519    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m2tdr_c8c74de0-5c52-4e2e-9f03-3b982a4e7667/konnectivity-agent/0.log"
Oct 30 04:19:58.410012 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:19:58.409959    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-130.ec2.internal_868aae1459799721b266e5e297e80105/haproxy/0.log"
Oct 30 04:20:04.996289 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:04.996255    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-557c8f8584-6lfw5_06f8e1b8-8ddd-4298-9c61-f326e03387a7/authorino/0.log"
Oct 30 04:20:06.972969 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:06.972928    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52ce43e0-2ca8-4559-9fcb-3ddad3b9e169/alertmanager/0.log"
Oct 30 04:20:07.095416 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.095388    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52ce43e0-2ca8-4559-9fcb-3ddad3b9e169/config-reloader/0.log"
Oct 30 04:20:07.213360 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.213330    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52ce43e0-2ca8-4559-9fcb-3ddad3b9e169/kube-rbac-proxy-web/0.log"
Oct 30 04:20:07.340300 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.340270    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52ce43e0-2ca8-4559-9fcb-3ddad3b9e169/kube-rbac-proxy/0.log"
Oct 30 04:20:07.459102 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.459065    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52ce43e0-2ca8-4559-9fcb-3ddad3b9e169/kube-rbac-proxy-metric/0.log"
Oct 30 04:20:07.578438 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.578408    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52ce43e0-2ca8-4559-9fcb-3ddad3b9e169/prom-label-proxy/0.log"
Oct 30 04:20:07.697246 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.697126    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_52ce43e0-2ca8-4559-9fcb-3ddad3b9e169/init-config-reloader/0.log"
Oct 30 04:20:07.826504 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.826463    4494 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"]
Oct 30 04:20:07.831139 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.831112    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:07.852108 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.852078    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"]
Oct 30 04:20:07.930003 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.929971    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lmk\" (UniqueName: \"kubernetes.io/projected/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-kube-api-access-45lmk\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:07.930179 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.930016    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-lib-modules\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:07.930179 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.930037    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-sys\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:07.930256 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.930182    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-proc\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:07.930256 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.930224    4494 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-podres\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:07.945741 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:07.945711    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z74j4_e41def15-5a16-4b6d-b79a-362c08010b6b/kube-state-metrics/0.log"
Oct 30 04:20:08.030742 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.030709    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-podres\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.031205 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.030751    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lmk\" (UniqueName: \"kubernetes.io/projected/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-kube-api-access-45lmk\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.031205 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.030842    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-podres\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.031205 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.030858    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-lib-modules\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.031205 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.030926    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-sys\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.031205 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.030961    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-lib-modules\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.031205 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.031024    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-sys\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.031205 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.031030    4494 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-proc\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.031205 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.031072    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-proc\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.054555 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.054522    4494 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lmk\" (UniqueName: \"kubernetes.io/projected/8bce57fa-4cf3-4719-a64f-4c2ae7eb649d-kube-api-access-45lmk\") pod \"perf-node-gather-daemonset-pknnt\" (UID: \"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d\") " pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.059755 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.059732    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z74j4_e41def15-5a16-4b6d-b79a-362c08010b6b/kube-rbac-proxy-main/0.log"
Oct 30 04:20:08.149782 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.149745    4494 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:08.181774 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.181744    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z74j4_e41def15-5a16-4b6d-b79a-362c08010b6b/kube-rbac-proxy-self/0.log"
Oct 30 04:20:08.488050 ip-10-0-130-130 kubenswrapper[4494]: W1030 04:20:08.488014    4494 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8bce57fa_4cf3_4719_a64f_4c2ae7eb649d.slice/crio-236c2cceedfe6409cf65c35742ab75233d4e935ce631d07d73642afba5e180cc WatchSource:0}: Error finding container 236c2cceedfe6409cf65c35742ab75233d4e935ce631d07d73642afba5e180cc: Status 404 returned error can't find the container with id 236c2cceedfe6409cf65c35742ab75233d4e935ce631d07d73642afba5e180cc
Oct 30 04:20:08.490325 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:08.490300    4494 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"]
Oct 30 04:20:09.315937 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:09.315877    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt" event={"ID":"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d","Type":"ContainerStarted","Data":"de85cd5aa72310ba29402be772c2b26873248e2b896e683941f9c09a93e510a4"}
Oct 30 04:20:09.316395 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:09.315957    4494 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt" event={"ID":"8bce57fa-4cf3-4719-a64f-4c2ae7eb649d","Type":"ContainerStarted","Data":"236c2cceedfe6409cf65c35742ab75233d4e935ce631d07d73642afba5e180cc"}
Oct 30 04:20:09.316395 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:09.316091    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:09.324741 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:09.324710    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sfkc2_8386b555-b07c-4481-9226-4a65739fd8db/node-exporter/0.log"
Oct 30 04:20:09.358450 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:09.358391    4494 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt" podStartSLOduration=2.358372084 podStartE2EDuration="2.358372084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:09.355796911 +0000 UTC m=+5073.561977130" watchObservedRunningTime="2026-10-30 04:20:09.358372084 +0000 UTC m=+5073.564552303"
Oct 30 04:20:09.443464 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:09.443434    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sfkc2_8386b555-b07c-4481-9226-4a65739fd8db/kube-rbac-proxy/0.log"
Oct 30 04:20:09.560655 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:09.560625    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sfkc2_8386b555-b07c-4481-9226-4a65739fd8db/init-textfile/0.log"
Oct 30 04:20:09.683682 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:09.683605    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dw86r_bf9aaba6-e684-4534-bbf9-4396fc97797a/kube-rbac-proxy-main/0.log"
Oct 30 04:20:09.802394 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:09.802365    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dw86r_bf9aaba6-e684-4534-bbf9-4396fc97797a/kube-rbac-proxy-self/0.log"
Oct 30 04:20:09.919785 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:09.919752    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dw86r_bf9aaba6-e684-4534-bbf9-4396fc97797a/openshift-state-metrics/0.log"
Oct 30 04:20:11.132874 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:11.132847    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-bw5fk_525f461c-2548-41c0-b83c-6bad98da93ad/prometheus-operator-admission-webhook/0.log"
Oct 30 04:20:11.258134 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:11.258108    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-745478bcb9-llz65_641b6d8c-11ea-4e24-9645-6c64e3b138ed/telemeter-client/0.log"
Oct 30 04:20:11.376842 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:11.376817    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-745478bcb9-llz65_641b6d8c-11ea-4e24-9645-6c64e3b138ed/reload/0.log"
Oct 30 04:20:11.498620 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:11.498547    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-745478bcb9-llz65_641b6d8c-11ea-4e24-9645-6c64e3b138ed/kube-rbac-proxy/0.log"
Oct 30 04:20:11.621507 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:11.621478    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7595945dc8-drcrc_d7f7009b-d19b-43ea-a7d1-1901c90daf79/thanos-query/0.log"
Oct 30 04:20:11.738712 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:11.738678    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7595945dc8-drcrc_d7f7009b-d19b-43ea-a7d1-1901c90daf79/kube-rbac-proxy-web/0.log"
Oct 30 04:20:11.859322 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:11.859288    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7595945dc8-drcrc_d7f7009b-d19b-43ea-a7d1-1901c90daf79/kube-rbac-proxy/0.log"
Oct 30 04:20:11.977050 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:11.977023    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7595945dc8-drcrc_d7f7009b-d19b-43ea-a7d1-1901c90daf79/prom-label-proxy/0.log"
Oct 30 04:20:12.095379 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:12.095346    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7595945dc8-drcrc_d7f7009b-d19b-43ea-a7d1-1901c90daf79/kube-rbac-proxy-rules/0.log"
Oct 30 04:20:12.215591 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:12.215509    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7595945dc8-drcrc_d7f7009b-d19b-43ea-a7d1-1901c90daf79/kube-rbac-proxy-metrics/0.log"
Oct 30 04:20:15.333513 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:15.333481    4494 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lgh6m/perf-node-gather-daemonset-pknnt"
Oct 30 04:20:15.655807 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:15.655730    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78dcd969cf-qqc8s_46d6ab97-46c3-4f46-ad7f-5f2bcd914fe1/console/0.log"
Oct 30 04:20:15.788595 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:15.788562    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-dvbg6_c2c612b6-be82-4c17-a26b-06c6768cae0f/download-server/0.log"
Oct 30 04:20:19.769858 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:19.769732    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zxwqk_bf0213f1-3092-4a53-ba3a-1cec86701275/dns/0.log"
Oct 30 04:20:19.885440 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:19.885404    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zxwqk_bf0213f1-3092-4a53-ba3a-1cec86701275/kube-rbac-proxy/0.log"
Oct 30 04:20:20.127039 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:20.127007    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vsx5v_ac850265-e643-4365-b318-3507b8c838b9/dns-node-resolver/0.log"
Oct 30 04:20:21.748548 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:21.748515    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f8mgk_33044703-7fec-4a2f-a1c7-cde0e8dd50bc/node-ca/0.log"
Oct 30 04:20:22.776603 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:22.776574    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7_d6653351-672f-4b10-bcca-68f6cfce20f6/istio-proxy/0.log"
Oct 30 04:20:22.842494 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:22.842464    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql_23cf63a6-e63a-4b06-9511-4d243425e553/istio-proxy/0.log"
Oct 30 04:20:23.032447 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:23.032353    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2_efe3c14d-9bd0-4153-a117-db52d9dae3ee/istio-proxy/0.log"
Oct 30 04:20:23.094654 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:23.094622    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-png5f_0d37115d-21f2-4744-9bbc-f86e83367b98/istio-proxy/0.log"
Oct 30 04:20:24.085281 ip-10-0-130-130 kubenswrapper[4494]: E1030 04:20:24.085245    4494 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pgwbh" containerName="extractor"
Oct 30 04:20:24.378796 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:24.378716    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7_d6653351-672f-4b10-bcca-68f6cfce20f6/istio-proxy/0.log"
Oct 30 04:20:24.500166 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:24.500132    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql_23cf63a6-e63a-4b06-9511-4d243425e553/istio-proxy/0.log"
Oct 30 04:20:24.869101 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:24.869070    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2_efe3c14d-9bd0-4153-a117-db52d9dae3ee/istio-proxy/0.log"
Oct 30 04:20:24.990819 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:24.990792    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-png5f_0d37115d-21f2-4744-9bbc-f86e83367b98/istio-proxy/0.log"
Oct 30 04:20:25.450502 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:25.450471    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-ff7k7_d6653351-672f-4b10-bcca-68f6cfce20f6/istio-proxy/0.log"
Oct 30 04:20:25.513102 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:25.513073    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-hjqql_23cf63a6-e63a-4b06-9511-4d243425e553/istio-proxy/0.log"
Oct 30 04:20:25.708128 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:25.708039    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-n8jq2_efe3c14d-9bd0-4153-a117-db52d9dae3ee/istio-proxy/0.log"
Oct 30 04:20:25.770910 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:25.770865    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5b4f8cc59b-png5f_0d37115d-21f2-4744-9bbc-f86e83367b98/istio-proxy/0.log"
Oct 30 04:20:26.545796 ip-10-0-130-130 kubenswrapper[4494]: I1030 04:20:26.545764    4494 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2d7p7_2c7ca2bd-121d-4ef9-ae00-4a139497c91d/serve-healthcheck-canary/0.log"